SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mohammad Jalali
Isfahan University of Technology
mjalali@ec.iut.ac.ir
Cheuk Ting Li
ctli@ie.cuhk.edu.hk
Farzan Farnia
farnia@cse.cuhk.edu.hk
The evaluation of generative models has received significant attention in the machine learning community. When applied to a multi-modal distribution which is common among image datasets, an intuitive evaluation criterion is the number of modes captured by the generative model. While several scores have been proposed to evaluate the quality and diversity of a model's generated data, the correspondence between existing scores and the number of modes in the distribution is unclear. In this work, we propose an information-theoretic diversity evaluation method for multi-modal underlying distributions. We utilize the Renyi Kernel Entropy (RKE) as an evaluation score based on quantum information theory to measure the number of modes in generated samples. To interpret the proposed evaluation method, we show that the RKE score can output the number of modes of a mixture of sub-Gaussian components. We also prove estimation error bounds for estimating the RKE score from limited data, suggesting a fast convergence of the empirical RKE score to the score for the underlying data distribution. Utilizing the RKE score, we conduct an extensive evaluation of state-of-the-art generative models over standard image datasets. The numerical results indicate that while the recent algorithms for training generative models manage to improve the mode-based diversity over the earlier architectures, they remain incapable of capturing the full diversity of real data. Our empirical results provide a ranking of widely-used generative models based on the RKE score of their generated samples<sup>1</sup>.
Deep generative models trained by generative adversarial networks (GANs) [1] and diffusion models [2] have achieved impressive results in various unsupervised learning settings [3, 4, 5, 6]. Due to their success in generating image samples with high visual quality, the analysis of large-scale generative models has received great attention in the machine learning community. In particular, the evaluation of generative models has been extensively studied in the recent literature to understand the benefits and drawbacks of existing approaches to training generative models.
To address the evaluation task for generative models, multiple assessment scores have been proposed in the literature. The existing evaluation metrics can be divided into two general categories: 1)
<sup>1</sup>The code repository is available at https://github.com/mjalali/renyi-kernel-entropy.
distance-based metrics including Fréchet Inception Distance (FID) [7] and Kernel Inception distance (KID) [8] scores, which measure a distance between the learned generative model and data distribution, 2) quality-based metrics including the Inception score [9], precision and recall scores [10, 11], and density and coverage scores [12], aiming to measure the quality and diversity of the generated samples based on the confidence and variety of labels assigned by a pre-trained neural net on ImageNet.
On the other hand, a different measure of diversity that is popular and standard in the information theory literature is entropy. The primary challenge with an entropy-based approach to the assessment of generative models is the statistical costs of entropy estimation in the typical high-dimensional spaces of image data, requiring a sample size exponentially growing with the dimension of data. Consequently, without further assumptions on the data distribution, it will be statistically and computationally infeasible to estimate the entropy value for high-dimensional image data.
In this work, we propose a novel information-theoretic approach for the diversity evaluation of generative models. Our proposed approach targets multi-modal data distributions comprised of several distinct modes, which is an applicable assumption to image datasets with a cluster-based structure due to their latent color and shape-based features. In this approach, we follow the entropy calculation in quantum information theory [13, 14] and utilize matrix-based Rényi entropy scores to evaluate the variety of samples produced by a generative model.
Considering the Rényi entropy in the Gaussian kernel space, we propose Rényi kernel entropy (RKE) and relative Rényi kernel entropy (RRKE) scores to measure the absolute and relative diversity of a multi-modal distribution with respect to the actual data distribution. We develop computationally efficient methods for estimating these entropy scores from empirical data with statistical convergence guarantees. We also prove that the RKE score will converge to standard differential entropy as the Gaussian kernel bandwidth approaches zero.
We provide an interpretation of the RKE score by deriving its closed-form expression for benchmark Gaussian mixture models (GMMs). In the GMM case, we show that the proposed RKE score reveals the number of Gaussian components in the underlying GMM, which motivates the method's application to general mixture distributions. We further extend this interpretation to mixture models with sub-Gaussian modes, e.g. modes with a bounded support set, to support the RKE score in more general settings. We discuss the numerical performance of the proposed score in synthetic mixture settings. Our numerical results demonstrate the fast convergence of the RKE score from a limited number of synthetic mixture data.
Next, we present the results of our numerical experiments on the evaluation of various state-of-the-art GAN and diffusion models using the RKE and RRKE metrics. Our numerical evaluation of the RKE score shows the lower diversity obtained by standard GAN models compared to real training data, which provides an information-theoretic numerical proof complementary to the birthday-paradox empirical proof for the same result in [15]. Furthermore, our empirical results suggest that the recent GAN and diffusion model architectures improve the mode-based variety of generated data over earlier GAN formulations. We can summarize the main contributions of this work as follows:
- Proposing an information theoretic approach to evaluate the diversity of generative models
- Developing computationally efficient methods to compute Rényi kernel entropy scores
- Providing theoretical and numerical support for the proposed evaluation methodology in the benchmark setting of Gaussian and sub-Gaussian mixture models
- Diversity evaluation of standard generative models using the information-theoretic method
The evaluation of GAN-based generative models has been studied by a large body of related works. As surveyed in [16], several evaluation methods have been developed in the literature. The Inception score (IS) [9] uses the output of a pre-trained Inception-net model as features and proposes a score summing up the entropy-based diversity of assigned labels' distribution and confidence score averaged over the conditional labels' distribution. The modified IS (m-IS) in [17] substitutes the KL-divergence term in IS with a cross entropy term, which helps m-IS capture diversity within images from a certain class. Unlike these works, our proposed approach bases on matrix-based entropy scores which capture the number of clusters in a multi-modal distribution.
Also, several distance-based evaluation metrics have been proposed in the deep learning literature. The Wasserstein Critic [18] attempts to approximate the Wasserstein distance between the real and generated samples. The Fréchet Inception Distance (FID) [7] measures a distance based on the embedding of the last layer of the pre-trained Inception-net, where it fits multivariate Gaussian models to real and generated data and calculates their Fréchet distance [19]. [20] conduct a comprehensive comparison of several standard GAN architectures based on IS and FID scores, discussing the similarities and differences of the GANs' performance. As another variant of FID, [21] suggest a bias-free estimation of FID using quasi-Monte Carlo integration. In another related work, [8] propose Kernel Inception Distance (KID) as the squared maximum mean discrepancy (MMD) between two distributions. Adversarial accuracy and divergence scores in [22] utilize two classifiers to compute the closeness of distributions of real and fake data conditioned on category labels from the two classifiers. Unlike the above metrics, our proposed relative Rényi entropy score focuses on the number of common modes between real and fake data, and hence reduces the statistical complexity of estimating the entropy.
The diversity vs. quality tradeoff of GANs' generated samples has also been studied in multiple related works. [15] examine the diversity of GANs' data through a birthday paradox-based approach and suggest that the support set size of GANs' data could be smaller than real training data. The precision and recall evaluation by [10] assigns a two-dimensional score where precision is defined as the portion of fake data that can be generated by real distribution while recall is defined as the portion of real data that can be generated by the generative model. The improved precision and recall in [11] further address the sensitivity disadvantages of these scores by estimating the density function via the k-nearest neighbour method. Also, the density and coverage scores [12] provide a more robust version of precision and recall metrics to outliers. Additionally, [23] proposed a 3-dimensional metric, that measures the fidelity, diversity and generalization of models. We note that our work offers a complementary approach to the diversity evaluation for GANs, and since it directly estimates the matrix-based entropy from data, it requires a comparatively smaller sample size for proper estimation.
3.1 Kernel-based Feature Maps and Representation
Throughout the paper, we use  $\mathbf{X} \in \mathcal{X}$  to denote the data vector. Also, we denote the kernel feature map by  $\phi: \mathbb{R}^t \to \mathbb{R}^d$  which gives us the kernel function  $k: \mathbb{R}^t \times \mathbb{R}^t \to \mathbb{R}$  as the inner product of the corresponding feature vectors:  $k(\mathbf{x}, \mathbf{y}) = \langle \phi(\mathbf{x}), \phi(\mathbf{y}) \rangle$ . Given  $n$  training data  $\mathbf{x}_1, \ldots, \mathbf{x}_n$  we use  $\Phi$  to denote the normalized kernel feature map-based data matrix, i.e.,
$$
\Phi = \frac {1}{\sqrt {n}} \left[ \begin{array}{c} \phi (\mathbf {x} _ {1}) \\ \vdots \\ \phi (\mathbf {x} _ {n}) \end{array} \right].
$$
Then, the kernel matrix  $K \in \mathbb{R}^{n \times n}$  whose  $(i,j)$ th entry will be  $\frac{1}{n} k(\mathbf{x}_i,\mathbf{x}_j)$  will be identical to  $K = \Phi \Phi^\top$ . Observe that, by definition,  $K$  will be a positive semi-definite (PSD) matrix, possessing positive eigenvalues  $\lambda_1,\ldots,\lambda_n$  where the non-zero eigenvalues are shared with the empirical kernel covariance matrix  $C = \Phi^\top \Phi$ . We call a kernel function normalized if  $k(\mathbf{x},\mathbf{x}) = 1$  for every  $\mathbf{x} \in \mathcal{X}$ . A standard example of a normalized kernel function is the Gaussian kernel with bandwidth parameter  $\sigma$  defined as:
$$
k _ {\sigma} (\mathbf {x}, \mathbf {y}) := \exp \Bigl (- \frac {\| \mathbf {x} - \mathbf {y} \| _ {2} ^ {2}}{2 \sigma^ {2}} \Bigr).
$$
For every normalized kernel function, the eigenvalues of kernel matrix  $K$  (and also empirical kernel covariance  $C$ ) will be non-negative and add up to 1 as the trace of  $K$  will be 1. As a result,  $K$ 's eigenvalues can be interpreted as a probability sequence.
3.2 Rényi Entropy for PSD Matrices
A standard extension of the entropy concept to PSD matrices is the matrix-based Rényi entropy [14]. The Rényi entropy of order  $\alpha > 0$  for a PSD matrix  $A \in \mathbb{R}^{d \times d}$  with eigenvalues  $\lambda_1, \ldots, \lambda_d$  is defined as
$$
\mathrm {R E} _ {\alpha} (A) := \frac {1}{1 - \alpha} \log \big (\operatorname {T r} (A ^ {\alpha}) \big) = \frac {1}{1 - \alpha} \log \left(\sum_ {i = 1} ^ {d} \lambda_ {i} ^ {\alpha}\right),
$$
where  $\mathrm{Tr}$  denotes the trace operator. A commonly-used special case which we use throughout the paper is the Rényi entropy of order  $\alpha = 2$  defined as  $\mathrm{RE}_2(A) = \log \left(1 / \sum_{i=1}^{d} \lambda_i^2\right)$ .
Proposition 1. For every PSD  $A \in \mathbb{R}^d$ , the following holds where  $\| \cdot \|_F$  denotes the Frobenius norm:
$$
\mathrm {R E} _ {2} (A) = \log \left(1 / \operatorname {T r} \left(A A ^ {\top}\right)\right) = \log \left(1 / \| A \| _ {F} ^ {2}\right).
$$
3.3 Relative Renyi Entropy
To measure the relative diversity of a matrix  $A \in \mathbb{R}^{d \times d}$  with respect to another matrix  $B \in \mathbb{R}^{d \times d}$ , one can use the sandwiched relative Rényi entropy of order  $\alpha$  [24] defined as
$$
\mathrm {R R E} _ {\alpha} (A, B) = \frac {1}{\alpha - 1} \log \left(\operatorname {T r} \left(\left(B ^ {\frac {1 - \alpha}{2 \alpha}} A B ^ {\frac {1 - \alpha}{2 \alpha}}\right) ^ {\alpha}\right)\right).
$$
A widely-used special case is the relative entropy of order  $\alpha = \frac{1}{2}$  which is commonly called the Fidelity score in quantum information theory. The definition of the Fidelity score is
$$
\mathrm {R R E} _ {1 / 2} (A, B) := - 2 \log \left(\operatorname {T r} \left(\sqrt {B ^ {1 / 2} A B ^ {1 / 2}}\right)\right).
$$
We note that the relative Rényi entropy of order  $\alpha = 2$  requires an invertible matrix  $B$  which may not hold in the applications to multi-modal distributions with rank-deficient kernel covariance matrices as discussed in the next sections.
4 A Diversity Metric for Multi-modal Distributions
4.1 Kernel-based Rényi Entropy Scores
Given the kernel matrix  $K$  computed using the data  $\mathbf{x}_1, \ldots, \mathbf{x}_n$  of random vector  $\mathbf{X}$ , the empirical Renyi kernel entropy (RKE) of the observed data can be defined as
$$
\widehat {\mathrm {R K E}} _ {\alpha} (\mathbf {X}) := \mathrm {R E} _ {\alpha} (K),
$$
which, as discussed in [14], is a diversity measure of the data. This diversity measurement approach will be interpreted and justified in the next subsection. Note that if the dimension  $d$  of the feature space is finite, since  $K = \Phi \Phi^{\top}$  and empirical covariance matrix  $\widehat{C} = \Phi^{\top}\Phi$  share the same eigenvalues, we have
$$
\widehat {\mathrm {R K E}} _ {\alpha} (\mathbf {X}) = \mathrm {R E} _ {\alpha} (\widehat {C}) = \mathrm {R E} _ {\alpha} \bigg (\frac {1}{n} \sum_ {i = 1} ^ {n} \phi (\mathbf {x} _ {i}) \phi (\mathbf {x} _ {i}) ^ {\top} \bigg).
$$
Therefore, we can see that the empirical Renyi kernel entropy  $\widehat{\mathrm{RKE}}_{\alpha}(\mathbf{X})$  is an estimate of the following quantity about the underlying distribution  $P_{X}$  where  $\mathbf{X}$  is sampled from, which we call the Renyi kernel entropy of  $P_{X}$ :
$$
\mathrm {R K E} _ {\alpha} (\mathbf {X}) := \mathrm {R E} _ {\alpha} (C _ {X}).
$$
Here  $C_X$  denotes the kernel covariance matrix of distribution  $P_X$  defined as
$$
C _ {X} := \mathbb {E} _ {P _ {X}} [ \phi (X) \phi (X) ^ {\top} ] = \int P _ {X} (x) \phi (x) \phi (x) ^ {\top} \mathrm {d} x.
$$
Similarly, we can use the kernel-based relative Rényi entropy as a measure of joint diversity between distributions  $P_{X}, P_{Y}$  of random vectors  $\mathbf{X}, \mathbf{Y}$ . Here, for random vectors  $\mathbf{X}, \mathbf{Y}$  distributed according to  $P_{X}, P_{Y}$ , we define the relative Rényi kernel entropy  $(RRKE_{\alpha})$  score as the order- $\alpha$  relative kernel entropy between their kernel covariance matrices  $C_{X}, C_{Y}$ :
$$
\mathrm {R R K E} _ {\alpha} (\mathbf {X}, \mathbf {Y}) = \mathrm {R R E} _ {\alpha} \left(C _ {X}, C _ {Y}\right).
$$
In order to estimate the above relative entropy score, we can use the empirical RRKE score between the empirical kernel covariance matrices for samples  $\mathbf{x}_1,\ldots ,\mathbf{x}_n$  from  $P_{X}$  and samples  $\mathbf{y}_1,\dots ,\mathbf{y}_m$  from  $P_{Y}$ :
$$
\widehat {\mathrm {R R E}} _ {\alpha} (\mathbf {X}, \mathbf {Y}) := \mathrm {R R E} _ {\alpha} \bigg (\frac {1}{n} \sum_ {i = 1} ^ {n} \phi (\mathbf {x} _ {i}) \phi (\mathbf {x} _ {i}) ^ {\top}, \frac {1}{m} \sum_ {j = 1} ^ {m} \phi (\mathbf {y} _ {j}) \phi (\mathbf {y} _ {j}) ^ {\top} \bigg).
$$
In the rest of this section, we show how the kernel-based Rényi entropy score evaluated under a Gaussian kernel can relate to the number of modes of multi-modal distributions with sub-Gaussian components, and subsequently how the relative Rényi entropy score counts the number of joint modes between two multi-modal distributions with sub-Gaussian modes.
4.2 RKE as a Measure of the Number of Modes
We consider a multi-modal underlying distribution  $P_{X}$  consisting of  $k$  distinct modes. Here, our goal is to show that the order-2 RKE score under the Gaussian kernel can count the number of present modes in the distribution. To this end, we analyze the order-2 Rényi kernel entropy score of the kernel covariance matrix  $C_{X}$  of mixtures of Gaussian and sub-Gaussian components and theoretically show that the RKE score reduces to the number of well-separated modes.
First, we derive the closed-form expression of the order-2 Rényi kernel entropy of a Gaussian mixture model under the Gaussian kernel. Here, we use the following notation to denote a  $k$ -component Gaussian mixture model where the  $i$ th component has frequency  $\omega_{i}$ , mean vector  $\pmb{\mu}_{i}$  and Covariance matrix  $\Sigma_{i}$ :  $P_{\mathrm{GMM}(\boldsymbol{\omega},\boldsymbol{\mu},\Sigma)} = \sum_{i=1}^{k}\omega_{i}\mathcal{N}(\pmb{\mu}_{i},\Sigma_{i})$ . Also, for a positive definite matrix  $A \in \mathbb{R}^{d\times d}$ , we use  $\|\cdot\|_A$  to denote the  $A$ -norm defined as  $\|\mathbf{x}\|_A = \sqrt{\mathbf{x}^\top A\mathbf{x}}$ .
Theorem 1. Suppose that the distribution of  $\mathbf{X}$  is given by the Gaussian mixture model  $P_{\mathrm{GMM}(\omega ,\mu ,\Sigma)}$  Then, order-2 Renyi kernel score under the Gaussian kernel with bandwidth  $\sigma$  denoted by  $G_{\sigma}$  , is
$$
\mathrm {R K E} _ {2} ^ {\mathrm {G} _ {\sigma}} (\mathbf {X}) = - \log \Bigg (\sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {k} \bigg [ \omega_ {i} \omega_ {j} e ^ {- \frac {\| \mu_ {i} - \mu_ {j} \| _ {A _ {i , j}} ^ {2}}{\sigma^ {2}}} \det \Big (I + \frac {2}{\sigma^ {2}} (\Sigma_ {i} + \Sigma_ {j}) \Big) ^ {- \frac {1}{2}} \bigg ] \Bigg),
$$
where  $A_{i,j}$  is defined as follows given the  $d \times d$  identity matrix  $I$ :
$$
\begin{array}{l} A _ {i, j} := I - \left(I + 2 \sigma^ {2} \Sigma_ {i} ^ {- 1} - \left(I + 2 \sigma^ {2} \Sigma_ {j} ^ {- 1}\right) ^ {- 1}\right) ^ {- 1} - \left(I + 2 \sigma^ {2} \Sigma_ {j} ^ {- 1} - \left(I + 2 \sigma^ {2} \Sigma_ {i} ^ {- 1}\right) ^ {- 1}\right) ^ {- 1} \\ + \left(2 \sigma^ {2} \Sigma_ {i} ^ {- 1} + 2 \sigma^ {2} \Sigma_ {j} ^ {- 1} + 4 \sigma^ {4} \Sigma_ {i} ^ {- 1} \Sigma_ {j} ^ {- 1}\right) ^ {- 1} + \left(2 \sigma^ {2} \Sigma_ {i} ^ {- 1} + 2 \sigma^ {2} \Sigma_ {j} ^ {- 1} + 4 \sigma^ {4} \Sigma_ {j} ^ {- 1} \Sigma_ {i} ^ {- 1}\right) ^ {- 1}. \\ \end{array}
$$
Proof. We defer the proof to the Appendix.
Corollary 1. Suppose  $\mathbf{X} \sim \mathrm{GMM}(\boldsymbol{\omega}, \boldsymbol{\mu}, \Sigma)$  follows from a Gaussian mixture model with isotropic covariance matrices  $\Sigma_{i} = \sigma_{i}^{2}I$ . Defining the coefficients  $a_{i,j} \coloneqq 1 + \frac{2(\sigma_i^2 + \sigma_j^2)}{\sigma^2}$ , the RKE score will be
$$
\mathrm {R K E} _ {2} ^ {\mathrm {G} _ {\sigma}} (\mathbf {X}) = - \log \bigg (\sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {k} \omega_ {i} \omega_ {j} a _ {i, j} ^ {- \frac {d}{2}} e ^ {- \frac {a _ {i , j} \| \boldsymbol {\mu} _ {i} - \boldsymbol {\mu} _ {j} \| _ {2} ^ {2}}{\sigma^ {2}}} \bigg).
$$
Theorem 1 and Corollary 1 show that if  $\sigma \gg \max_{i} \| \Sigma_{i} \|_{\mathrm{sp}}$ , i.e. the kernel bandwidth dominates the spectral norm (maximum eigenvalue) of the component-wise covariance matrices, then  $A_{i,j} \approx I$  and  $a_{i,j} \approx 1$  and the RKE score will approximately be
$$
\mathrm {R K E} _ {2} ^ {\mathrm {G} _ {\sigma}} (\mathbf {X}) \approx - \log \Bigl (\sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {k} \omega_ {i} \omega_ {j} e ^ {- \frac {\| \mu_ {i} - \mu_ {j} \| _ {2} ^ {2}}{\sigma^ {2}}} \Bigr).
$$
The next theorem shows the above approximation generalizes to mixtures of sub-Gaussian components, e.g. modes with bounded support sets, and also provides an approximation error bound.
Theorem 2. Suppose that  $\mathbf{X} \in \mathbb{R}^d$  has a mixture distribution  $P_{\mathrm{SGMM}} = \sum_{i=1}^{k} \omega_i P_i$  where the  $k$ th component occurs with probability  $\omega_k$  and has a sub-Gaussian distribution with parameter  $\sigma_i$ , i.e., its moment-generating function (MGF)  $M_{P_i}$  satisfies the following for every vector  $\beta \in \mathbb{R}^d$  given the mean vector  $\pmb{\mu}_i$  for  $P_i$ :  $\mathbb{E}_{P_i}[\exp(\beta^\top (\mathbf{X} - \pmb{\mu}_i))] \leq \exp(\|\beta\|_2^2 \sigma_i^2 / 2)$ . Then, the following approximation error bound holds where we define  $\alpha_i^2 := 1 + 2\sigma_i^2 / \sigma^2$ :
$$
\left| \exp \left(- \mathrm {R K E} _ {2} ^ {\mathrm {G} _ {\sigma}} (\mathbf {X})\right) - \sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {k} \omega_ {i} \omega_ {j} e ^ {- \frac {\| \boldsymbol {\mu} _ {i} - \boldsymbol {\mu} _ {j} \| _ {2} ^ {2}}{\sigma^ {2}}} \right| \leq \sqrt {\sum_ {i = 1} ^ {k} 8 \omega_ {i} \big (1 - \alpha_ {i} ^ {- d} \big)}.
$$
More generally, we show that the RKE under a Gaussian kernel with a small bandwidth provides an estimate of the differential Rényi entropy, which connects the approach with a smoothed differential entropy estimator in high-dimensional settings.
Theorem 3. Suppose that the underlying distribution of  $X$  has a continuous probability density function  $P_{X}$ . The order-2 Rényi kernel score under Gaussian kernel with bandwidth  $\sigma$  for this underlying distribution satisfies
$$
\lim  _ {\sigma \rightarrow 0} \left(\mathrm {R K E} _ {2} ^ {\mathrm {G} _ {\sigma}} (\mathbf {X}) + \frac {d}{2} \log (\pi \sigma^ {2})\right) = - \log \left(\int P _ {X} (\mathbf {x}) ^ {2} \mathrm {d} \mathbf {x}\right),
$$
where the right hand side is the order-2 Rényi differential entropy of  $P_{X}$ .
4.3 RRKE as the Number of Common Modes
In order to measure the joint mode-based diversity, we propose the order-  $\frac{1}{2}$  relative Renyi entropy score. We note that our choice of order  $\frac{1}{2}$  is due to the existing inverse covariance matrix term in the relative entropies of orders greater than 1, which is not applicable to rank deficient matrices expected in the case of well-separated multi-modal distributions. Furthermore, the order  $\frac{1}{2}$  relative entropy, well-known as the fidelity score, is a commonly-used and well-analyzed relative entropy case in quantum information theory.
To interpret the application of the order-  $\frac{1}{2}$  relative entropy, we show the following theorem discussing how the negative RRKE score could approximate the joint diversity between two input multimodal distributions with sub-Gaussian components.
Theorem 4. Suppose that  $\mathbf{X},\mathbf{Y}\in \mathbb{R}^d$  are random vectors with mixture distributions  $P_{\mathrm{MM}_1} = \sum_{i = 1}^{k}\omega_iP_i$  and  $P_{\mathrm{MM_2}} = \sum_{i = 1}^{k}\eta_iQ_i$ , respectively, where  $\omega_{i},\eta_{i}$  denote the frequency of the ith component. We also assume that  $P_{i}$  and  $Q_{i}$  have mean vectors  $\pmb{\mu}_{i}$  and  $\zeta_{i}$  respectively and are both  $\sigma_{i}$ -sub-Gaussian. Then, the following approximation error bound holds for order- $\frac{1}{2}$  relative Renyi entropy where we define  $\alpha_{i}^{2} = 1 + 2\sigma_{i}^{2} / \sigma^{2}$ :
$$
\left| \exp \left(- \mathrm {R R K E} _ {2} ^ {\mathrm {G} _ {\sigma}} (\mathbf {X}, \mathbf {Y})\right) - \sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {k} \sqrt {\omega_ {i} \eta_ {j}} e ^ {- \frac {\| \mu_ {i} - \xi_ {j} \| _ {2} ^ {2}}{\sigma^ {2}}} \right| \leq \sqrt [ 4 ]{\sum_ {i = 1} ^ {k} 3 2 \left(\omega_ {i} + \eta_ {i}\right) \left(1 - \alpha_ {i} ^ {- d}\right)}
$$