An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions
Authors: Mohammad Jalali
Cheuk Ting Li
Farzan Farnia
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative Models; Evaluation in Learning; Information Measures

Abstract: The evaluation of generative models has received significant attention in the machine learning community.  When applied to a multi-modal distribution which is common among image datasets, an intuitive evaluation criterion is the number of modes captured by the generative model. While several scores have been proposed to evaluate the quality and diversity of a model's generated data, the correspondence between existing scores and the number of modes in the distribution is unclear. In this work, we propose an information-theoretic diversity evaluation method for multi-modal underlying distributions. We utilize the R\'enyi Kernel Entropy (RKE) as an evaluation score based on quantum information theory to measure the number of modes in generated samples. To interpret the proposed evaluation method, we show that the RKE score can output the number of modes of a mixture of sub-Gaussian components. We also prove estimation error bounds for estimating the RKE score from limited data, suggesting a fast convergence of the empirical RKE score to the score for the underlying data distribution. Utilizing the RKE score, we conduct an extensive evaluation of state-of-the-art generative models over standard image datasets. The numerical results indicate that while the recent algorithms for training generative models manage to improve the mode-based diversity over the earlier architectures, they remain incapable of capturing the full diversity of real data. Our empirical results provide a ranking of widely-used generative models based on the RKE score of their generated samples.

An Information-Theoret Evaluation of Generative Models in Learning Multi-modal Distributions
Mohammad Jalali
Cheuk Ting Li
Department of Electrical and Computer Engineering Isfahan University of Technology mjalalise fecalliec....
Department of Information Engineering The Chinese University of Hong Kong ctli@ie. cuhk. edu .kk
Farzan Farnia
Department of Computer Science and Engineering The Chinese University of Hong Kong farni farniacsse.o cuhk. edu. hk
Abstract
The evaluation of generative models has received significant attention in the ma- chine learning community. When applied multi-modal distribution which is common among image datasets, an intuitive evaluation criterion is the number modes captured by the generative model. While several scores have been proposed to evaluate the quality and diversity of a model's generated data, the correspondence between existing scores and the number of modes in the distribution is unclear. In this work, we propose an information-thoorttee diversity evaluation method for multi- modal underlying distributions. We utilize the Rényi Kernel Entropy (RKE) as an evaluation score based on quantum information theory to measure the number of modes in generated samples. To interpret the proposed evaluation method, we show that the RKE score can output the number of modes of a mixture of sub- Gaussian components. We also prove estimation error bounds for estimating the RKE score from limited data, suggesting a fast convergence of the empirical RKE score the score for the underlying data distribution. Utilizing the RKE score, we conduct an extensive evaluation of state-of-the-ate generative models over standard image datasets. The numerical results indicate that while the recent algorithms for training generative models manage to improve the mode-based diversity over the earlier architectures, they remain incapable of capturing the full diversity of real data. Our empirical results provide a ranking of widely-used generative models based on the RKE score of their generated samples[
Introduction
Deep generative models trained by generative adversarial networks (GANs) [ and diffusion models
2 have achieved impressive results in various unsupervised learning settings B 5 6 Due to their success in generating image samples with high visual quality, the analysis of large-scale generative models has received great attention in the machine learning community. In particular, the evaluation of generative models has been extensively studied in the recent literature to understand the benefits and drawbacks of existing approaches training generative models.
To address the evaluation task for generative models, multiple assessment scores have been proposed in the literature. The existing evaluation metrics can be divided into two general categories: 1)
The code repository is available at http://///ghthh.o.nooaaaaaaaa/aa
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

bastance-based metrics including Fréchet Inception Distance (FID) 7 and Kernel Inception distance (KID) scores, which measure distance between the learned generative model and data distribution. 2) quality-based metrics including the Inception score [9, precision and recall scores [0] 11 and
density and coverage scores [2], aiming to measure the quality and diversity of the generated samples based on the confidence and variety of labels assigned by pre-trained neural net on ImageNet.
On the other hand, different measure of diversity that is popular and standard in the information theory literature is entropy. The primary challenge with an entropy-based approach to the assessment of generative models is the statistical costs of entropy estimation the typical high-dimensional spaces of image data, requiring a sample size exponentially growing with the dimension of data. Consequently, without further assumptions on the data distribution, will be statistically and computationally infeasible to estimate the entropy value for high-ding -igh-dimensional image data.
In this work, we propose a novel information-theoreti approach for the diversity evaluation of generative models. Our proposed approach targets multi-modal data distributions comprised of several distinct modes, which an applicable assumption image datasets with cluster-based structure due to their latent color and shape- based features. In this approach, we follow the entropy calculation in quantum information theory [33 [4 and utilize matrix-based Rényi entropy scores to evaluate the variety of samples produced by generative model.
Considering the Rényi entropy in the Gaussian kernel space, we propose Rényi kernel entropy (RKE) and relative Rényi kernel entropy (RRKE) scores to measure the absolute and relative diversity of a multi-modal distribution with respect the actual data distribution. We develop computationally efficient methods for estimating these entropy scores from empirical data with statistical convergence guarantees. We also prove that the RKE score will converge to standard differential entropy as the Gaussian kernel bandwidth approaches zero
We provide an interpretation of the RKE score by deriving its closed-form expression for benchmark Gaussian mixture models (GMMs). In the GMM case, we show that the proposed RKE score reveals the number of Gaussian components in the underlying GMM, which motivates the method's application to general mixture distributions. We further extend this interpretation to mixture models with sub-Gaussian modes, e.g. modes with bounded support set, to support the RKE score in more general settings. We discuss the numerical performance of the proposed score in synthetic mixture settings. Our numerical results demonstrate the fast convergence of the RKE score from limited number of synthetic mixture data.
Next, we present the results of our numerical experiments the evaluation of various state-of-the-art GAN and diffusion models using the RKE and RRKE metrics. Our numerical evaluation of the RKE score shows the lower diversity obtained by standard GAN models compared to real training data, which provides an information theoretic numerical proof complementary to the birthday-paradee empirical proof for the same result in [15). Furthermore, our empirical results suggest that the recent GAN and diffusion model architectures improve the mode-based variety of generated data over earlier GAN formulations. We can summarize the main contributions of this work as follows:
Proposing an information theoretic approach to evaluate the diversity of generative models
Developing computationally efficient methods to compute Rényi kernel entropy scores
Providing theoretical and numerical support for the proposed evaluation methodology in the benchmark setting of Gaussian and sub-Gaussian mixture models
Diversity evaluation of standard generative models using the information-thhoortio method
2 Related Work
The evaluation of GAN- generative models has been studied by a large body of related works. As surveyed in [6]. several evaluation methods have been developed in the literature. The Inception score (IS) 9 uses the output of a pre-trained Inception-ne model as features and proposes a score summing up the entropy- based diversity of assigned labels' distribution and confidence score averaged over the conditional labels' distribution. The modified (m-IS) in [17 substitutes the KL-ddivergence term in IS with a cross entropy term, which helps m-IS capture diversity within images from certain class. Unlike these works, our proposed approach bases on matrix- based entropy scores which capture the number of clusters in a multi-moda distribution.

---

Also, several distance- based evaluation metrics have been proposed in the deep learning literature. The Wasserstein Critic [T] attempts to approximate the Wasserstein distance between the real and generated samples. The Fréchet Inception Distance (FID) [ measures distance based on the embedding of the last layer of the pre-trained Inception-net, where fits multivariate Gaussian models to real and generated data and calculates their Fréchet distance [9). (20] conduct comprehensive comparison of several standard GAN architectures based on IS and FID scores, discussing the similarities and differences of the GANs' performance. As another variant of FID, [21] suggest bias-free estimation FID using quasi-Monte Carlo integration. In another related work, g propose Kerne Inception Distance (KID) as the squared maximum mean discrepancy (MMD) between two distributions. Adversarial accuracy and divergence scores in [22 utilize two classifiers to compute the closeness of distributions of real and fake data conditioned classifiers. Unlike the above metrics, our proposed relative Rényi entropy score focuses on the number of common modes between real and fake data, and hence reduces the statistical complexity
category labels from the two
of estimating the entropy.
The diversity vs. quality tradcoff of GANs' generated samples has also been studied in multiple related works. [13] examine the diversity of GANs' data through birthday paradox-based approach and suggest that the support set size of GANs' data could be smaller than real training data. The precision and recall evaluation by [0] assigns a wvo-dimensional score where precision is defined as the portion of fake data that can be generated by real distribution while recall is defined the portion of real data that can be generated by the generative model. The improved precision and recall in [[]] further address the sensitivity disadvantages of these scores by estimating the density function via the k-nearest neighbour method. Also, the density and coverage scores [12 provide a more robust version of precision and recall metrics outliers. Additionally, (23) proposed 3-dimensional metric, that measures the fidelity, diversity and generalization of models. We note that our work offers a complementary approach to the diversity evaluation for GANs, and since it directly estimates the matrix-based entropy from data, it requires a comparatively smaller sample size for proper estimation.
Preliminaries
3.1 Kernel-based Feature Maps and Representation
Throughout the paper, we use X X' denote the data vector. Also, we denote the kernel feature map by Φ R - Rd which gives us the kernel function RR x R - IR as the inner product of the corresponding feature vectors: k(xxy) y) = (Ø(x), p(y)). Given n training data xI,-..;X, to denote the normalized kernel feature map-based data matrix, i.e.,
X,, we use
(x1)
6 (xn)
Then, the kernel matrix K paxn whose j) entry will be , x,) will be identical to n K = T Observe that, by definition, will be a positive semi-definite (PSD) matrix, possessing positive cigenvalues X1, A...... A, where the non-zero eigenvalues are shared with the empirical kernel covariance matrix C = TΦ. &b We call a kernel function normalized if k(x, x) = for every x E X. A standard example of normalized kernel function is the Gaussian kernel with bandwidth parameter
σ defined as:
(x, == exp Xx y13
20
For every normalized kernel function, the eigenvalues of kernel matrix K (and also empirical kernel covariance C' will be non-negative and add up to 1 as the trace of K will be As a result, K's cigenvalues can be interpreted probability sequence.
3.2 Rényi Entropy for PSD Matrices
A standard extension of the entropy concept to PSD matrices is the matrix- based Rényi entropy 14]. The Rényi entropy of order a > 0 for PSD matrix A E R with eigenvalues A Ad is defined as
d
1
RE, (A) == 1 a
log (Tr()) = a
log
a
3

---

where Tr denotes the trace operator. A commonly-used special case which we use throughout the paper is the Rényi entropy of order
defined (E2(A) log(1/ Ei=
Proposition 1. For every PSD A E R the following holds where || || denotes the Frobenius norm:
RE2 (4) = (og(1/T(AAAA) = loog(1 (og(1/||A|]))
3.3 Relative Rényi Entropy
To measure the relative diversity of a matrix A E IRdxd with respect to another matrix B E IRdxd one can use the sandwiched relative Rényi entropy of order [24] defined as
RRE (A, B) a-1 log Tr ( AB
widely-used special case is the relative entropy order a = which commonly called the Fidelity score in quantum information theory. The definition of the Fidelity score is
RRE1 (A, B3 == log (Tr(v (Tr(VB/22 /1/2A/////)
We note that the relative Rényi entropy of order a 2 requires an invertible matrix B which may not hold in the applications multi multi-modal modal distributions with rank-deficient kernel covariance matrices as discussed in the next sections.
A Diversity Metric for Multi-modal Distributions
4.1 Kernel-based Rényi Entropy Scores
Given the kernel matrix K computed using the data X1; ... X, of random vector X, the empirical Rényi kernel entropy (RKE) of the observed data can be defined as
RKE, (X) RE, (K),
which, as discussed 11,
diversity measure the data. This diversity measurement approach will be interpreted and justified in the next subsection. Note that if the dimension d of the feature space
is finite, since K Φ67 and empirical covariance matrix Õ = T & share the same eigenvalues, we have
RKE, (X) (C. = REA (2 "' oxx;) ((x)))())))
Therefore, we can see that the empirical Rényi kernel entropy RKE, (X) is an estimate of the following quantity about the underlying distribution Px where X is sampled from, which we call the Rényi kernel entropy of Px:
RKEE (X) = (Cx)
Here Cx denotes the kernel covariance matrix of distribution Px defined as
Cx = Epx [o(X)o(x)
Px(=))(=)((z) dzz
Similarly, we can use the kernel-based relative Rényi entropy as measure of joint diversity between distributions Px, Py of random vectors X, Y. Here, for random vectors X, Y distributed according to Px, Py, we define the relative Rényi kernel entropy (RRKE,) score as the order-a relative kernel entropy between their kernel covariance matrices Cx, Cy:
RRKE, [X, Y) RRE, (Cx,Cr).
In order to estimate the above relative entropy score, we can use the empirical RRKE score between empirical kernel covariance matrices for samples XI,..., Xn from Px and samples yı, ,,.....yyv. y from Py:
RRKE, X.Y = RRE, 1
L ==1
1 - Σ ol
o( )o(x))
In the rest of this section, we show how the kernel-based Rényi entropy score evaluated under Gaussian kernel can relate to the number of modes of multi-modal distributions with sub-Gaussian components, subsequently how the relative Rényi entropy score counts the number of joint modes between two multi-modal distributions with sub- Gao--Gassiin modes.

---

4.2 RKE as a Measure of the Number of Modes
We consider multi-moda underlying distribution Px consisting distinct modes. Here, our goal is to show that the order-2 RKE score under the Gaussian kernel can count the number of present modes the distribution. To this end, we analyze the order-2 Rényi kernel entropy score of the kernel covariance matrix Cx of mixtures of Gaussian and sub-Gaussing components and theoretically show that the RKE score reduces the number of well-separated modes.
First, we derive the closed-form expression of the order-2 Rényi kernel entropy of a Gaussian mixture model under the Gaussian kernel. Here, we use the following notation denote k-component Gaussian mixture model where the ith component has frequency wi, mean vector μ4 and Covariance matrix Σ3 PGMM(W,
E (H, 2,). Also, for a positive definite matrix A ddd use || ll to denote the A-norm defined as ||x]| vx Ax.
Theorem Suppose that the distribution of X given by the Gaussian mixture model PaMMAN,)))) Then, order-2 Rényi kernel score under the Gaussian kernel with bandwidth a, denoted by Go. is
4 ΣΣ
1
det (T 1+2200 (Z+++,))) -1) +)
RKF (X) -- log
wawee
where As, is defined as follows given the x dl identity matrix I:
Asg = I -I+ 20"200001000 +I+ 20 1)- (I 20 (1 20 (1+20' 1)
+20 + 22
+ 40 x ズ
+ (20 20 13 + 40 -
Proof. We defer the proof to the Appendix.
Corollary 1. Suppose X ~ GMM(ww, μ. follows from a Gaussian mixture model with isotropic
2(a?-a?)
covariance matrices Σ; = o? I. Defining the coefficients 0i,j == 1 1
the RKE score will be a²
wawja, 4 or lle, 1
RKE (x) log
Theorem and Corollary show that if o >> max; [EEllsp i.e. the kernel bandwidth dominates the spectral norm (maximum eigenvalue) of the component-wist wise covariance matrices, then A.,j Rs / and asjj and the RKE score will approximately be
RKEE (X) R - log
wiwze
i=1 1=1
The next theorem shows the above approximation generalizes to mixtures of sub-Gaussian compo- nents, e.g. modes with bounded support sets, and also provides approximation error bound.
Theorem 2. Suppose that X E & has a mixture distribution PSUMM = Ek= WiP; where the kth component occurs with probability Wh. and has sub- Lub-Gaussian distribution with parameter a. i.e. its coment-ggenraainn given the mean vector μt, for P.: Ep, [exp(3 (X -x4))) S exp exp(1111]00?/2)) Then, the following
function (MGF) Mp, satisfies the following for every vector B3 E Rd
approximation error bound holds where we define of := 1 20?
k
1m4 1m-1:06
4
exp (-RKES (x)
Sws (1 a. a;
Wiwiie
i=1Ø=1
Proof. We defer the proof to the Appendix.
More generally, we show that the RKE under a Gaussian kernel with a small bandwidth provides an estimate of the differential Rényi entropy, which connects the approach with smoothed differential entropy estimator in high digeenimensio settings.
5

---

Theorem 3. Suppose that the undertying distribution of X has continuous probability density function Px. The order-2 Rényi kernel score under Gaussian kernel with bandwidth for this underlying distribution satisfies
lim 2 (X) + d log(no²)) -olog (ʃ Px(x))dx)) a-0
2
where the right hand side is the order-2 Rényi differential entropy of Px.
Proof. We defer the proof to the Appendix.
4.3 RRKE as the Number of Common Modes
In order to measure the joint mode-basee diversity, we propose the order- relative Rényi entropy score. We note that our choice of order 3 is due to the existing inverse covariance matrix term in the relative entropies of orders greater than I, which is not applicable rank deficient matrices expected in the case of well-separated multi-modal distributions. Furthermore, the order ğ relative entropy. well-known as the fidelity score, is commonly-use and well-analyzed relative entropy case in quantum information theory.
To interpret the application of the order- & relative entropy, show the following theorem discussing how the negative RRKE score could approximate the joint diversity between two input multimodal distributions with sub-Gaussian components.
Theorem 4. Suppose that X, Y E Id are random vectors with mixture distributions PMM- E w,P, and PaM.- E n.Qu. respectively, where Wi, Th denote the frequency of the ith component. We also assume that P; and Q; have mean vectors μi; and S; respectively and are both --sub-Gaausicn Then, the following approximation error bound holds for order- relative Rényi entropy where we define of = 20-20,/00 /02:
RRKEE (X.EEEO(XXYY))))
(X,Y)
ΣΣ
wime 1
32 i + +n) at
exp
Proof. We defer the proof to the Appendix.
The above theorem shows that if the ith mode of mixture distribution Pmm, and the jthh mode of mixture distribution Pmmn sufficiently close, then they add nearly /welly the RRKE score.
5 Estimation Procedure and Guarantees
As discussed earlier, the RKE and RRKE scores for proper kernel function provide measures the absolute and relative mode-based diversity of multi-modal distributions. Here, we propose kernel- based estimators for these entropy scores. Our estimators suggest computationally and statistically feasible ways for approximating the entropy measures by exploiting the connections between the kernel similarity values and the Rényi entropy scores. In addition, we provide asyptoiii estimation error bounds for the proposed estimators analyze their sample complexity. Regarding the RKE score, the following results connect this score with the kernel function k(x, x').
Theorem Given a random vector X distributed as Px, the order-2 RKE score the result of the following equation, where X, X' are I1I) draws of Px:
RKEa -o (E UP
[K2 [22(X.
For samples X1.... NN, and Kxx (kxxx,,x)))) )nxn being the normalized kernel matrix for the observed samples, the empirical order-2 Kernel entropy is given by
Corollary 2.
(XE, = =- -lo
( [
Proof. We defer the proof to the Appendix. We note that the result of Corollary 2 for the empirical RKE score has already been shown and discussed in [4]

---

Vanilla GAN
ISGAN
WGAN-WC:
WGAN-GP
RKE-MC: 3,97, RRKE 183
RKE-MC: 2.10, RRKE: 2.34
RKE-ME 26.49. RRKE: 0.06
RKE-MC: 25.17. RRKE 0.01
Figure 1: GANs' generated samples for Gaussian mixtures and the RKE-MC and RRKE & scores.
As the above results suggest, we can use the normalized Frobenius norm of the kernel matrix to estimate the RKE score. Therefore, the computation of order-2 RKE only requires the Frobenius norm the kernel matrix, which can be computed in (n?²) complexity. We note that for a general order-a Rényi entropy,
apply the randomized algorithm in [25] for the computation of the order-a RKE score. Our next result bounds the estimation error for the empirical RKE score.
Theorem 6. Consider normalized kernel function satisfying k(x, x) 1 for every E X. Then, for every > the following bound will hold with probability at least 1 6:
< log
exp(( (xxp(-RKF2(X)) (X)) - -exp(-RKE2
Proof. We defer the proof to the Appendix.
As implied by the above results, the order-2 Rényi kernel entropy can be efficiently estimated from training data and the probability of an e-large eπror will exponentially diminish with the sample size n. Next, we discuss the computation approach for the order- & RRKE score. The following result reveals the kernel-based representation of this score in the empirical case.
Theorem 7, Consider empirical samples X1,..., Xn drawn drawnfrom Px and ,,...., drawn from Py. Then, the following identity holds for their empirical order- RRKE score:
RRKE (X,KE = - (oIxXl (1:)
where Kxy = - k(xx, Yi) nxm. denotes the normalized cross kernel matrix and || || denotes the nuclear norm, i.e. the sum of a matrix's 's singular values.
Proof. We defer the proof to the Appendix.
As implied by the above theorem, the RRKE score can be computed using the singular value decomposition (SVD) for finding the singular values of KXY. Note that the application of SVD requires (min (m²n, mn²)) computations given n, m samples from Px and Py, respectively. Therefore, computing the RRKE score could be more expensive than the RKE score, since the nuclear norm is more costly to compute than the Frobenius norm.
Numerical Results
We tested the performance of the proposed entropy-based diversity evaluation approach on several combinations of standard GAN architectures and datasets. Specifically, we used the synthetic component and 25-c2mm 25-component Gaussian mixture datasets in [26) and the following image datasets:

---

Table 1: Evaluated scores for three image datasets. RKE-MC (Mode Count) denotes exp (RKE).
Method
1
FID
Precision 1
Recall f
Density t
Coverage t RRKE
RKE-MC
Dataset
11.57
39.58
NVAE
5.85
51.67
0.36
0.50
0.60
2.01
17.65
VDVAE
10.51
37.51
0.34
0.78
0.23
0.21
1.81
32.49
CIFAR-IN 10
DCGAN
5.75
54.30
0.59
0.25
0.49
0.23
0.98
10.19
WGAN-WC
2.59
157.26
0.36
0.03
WGAN-GP
7.51
21.66
0,62
0.56
0.57
0.51
0.74
19.07
SAGAN
8.62
10.17
0,68
0.62
0.73
0.73
0.65
24.46
SNGAN
8.81
9.23
0.70
0.62
0.77
0.74
0.62
25.83
ContraGAN
9.69
4.02
0.75
0.62
0.99
0.86
0.52
29.80
ImngeNer
Dataset
33,99
155.86
SAGAN
8.21
46.98
0.55
0.49
0.44
0.27
1.42
25.68
SNGAN
8.12
48.96
0.55
0.46
0.40
0.26
1.46
27.18
BigGAN
11.57
27.34
0.60
0.58
0.53
0.43
1.23
39.61
ContraGAN
13.79
21.36
0.54
0.54
0.45
56.94
Dataset
357.35
1823.52
SAGAN-256
29.67
44.66
0.57
0.35
2.34
105.57
SNGAN-256
31.92
35.75
0.54
0.64
0.41
0.38
2.22
115.62
ContrGGAN-256
24.91
34.79
0,67
0.51
0.64
0.33
2.54
152.89
ImngeNet
BigGAN-256
28.33
33.48
0.58
0.37
2.28
106.07
ReACGAN-256
52.53
15.65
0,74
0.42
0.79
0.41
2.15
119.76
BigGAN-2048
96.42
4.49
0.71
0.58
0.80
0.65
1.83
606.18
StyleGAN-XL
204.73
1.94
0.77
0.61
0.67
0.81
1.50
1375.17
LDM-4G
242.62
3.60
0,86
0.60
0.69
0.78
1.56
1321.24
ADM-G
188.70
3.86
0.82
0.64
0.66
0.82
1.47
1407.75
CIFAR- 10 27.. Tiny-ImageNet (28). MS-COCO [29]. AFHQ 300, FFHQ [3 and ImageNet
32
We evaluated the performance the following list of widely-used VAE, GAN and diffusion
model architectures: NVAE (33. Very Deep VAE (VDVAE) (34), Vanilla GAN M. LSGAN [35),
Wassesrtein GAN with weight clipping (WGAN-WC) [8]. Wassesrtein GAN with gradient penalty
(WGAN-GP) (26]. DCGAN (6). Self-Attention GAN (SAGAN) (37) Spectrally-Normalizze GAN
(SNGAN) 38), ContraGAN 39), ReACGAN 40), BigGAN B. StyleGAN3 [41], StyleGAN-XI
42 GigaGAN 43 LDM
44 ADM-G 45 and BK-SDM 46. To have fair evaluation of the
models, we downloaded the trained generative models from the StudioGAN repository [47].
In our evaluation of generative models, we compared the performance of order-2 Rényi Kernel Entropy Mode Count (RKF-MC), defined as exp (xp(RE22(XX) and order- Relative Rényi Kernel Entropy (RRKE) with the following standard baselines widely used in the evaluation of generative models: Inception Score (IS) [9, Fréchet Inception Distance (FID) [7], Kernel Inception Distance (KID) precision and recall
density and coverage [12
To compute the RKE and RRKE scores for the Gaussian mixture cases, we measured the scores based on the output of the trained generator. For image datasets, we followed the standard approach in the literature and evaluated the scores for the representation of the generator's 's output characterized by an Inception-net-V3 model pre-trained on ImageNet. We note that the Inception-net -eettiee--et--aeedd evaluation methodology is consistent with the baseline methods. Also, to select bandwidth parameter o for the Gaussian kernel in the RKE and RRKE scores, we performed cross-validation and chose the smallest bandwidth a for which the reported score's 's standard deviation across 5,000 validation samples is below 0.01. Note that if the kernel bandwidth becomes overly small, the RKE score will grow almost logarithmically with the number of samples, and the standard deviation of its exponential, i.c. RKE mode count (RKE-MC), will increase almost linearly with the sample size and thus suffer from a large variance across disjoint sample sets. We provide a more detailed discussion of the bandwidth parameter's selection and the resulting variance in the Appendix.
Diversity Evaluation for Synthetic Mixture Datasets.
measured the RKE-MC and RRKE scores for the 8 and 25 component Gaussian mixture datasets in (26). Figure shows the real samples blue and GANs" generated samples red. the evaluated scores suggest, the RKF-MC scores managed to count the number of captured modes and the RRKE relative distance increased under

---

50
40
RKE-MC MC
30
20
10
AFHQ Dataset
FFHQ Dataset
0 0.0
0.2
0.4
0.6
0.8
1.0
V = 0.0
V = 0.3
V 0.6
== 1.0
Truncation V
(a) AFHQ generated samples with different v).
(b) RKE Mode Count for StyleGAN generated samples AFHQ and FFHQ varying with truncation factor
V = 0.2
V 0.4
V 0.6
V 0.8
V 1.0
(c) FFHQ generated samples with different vb.
Figure 2: RKE mode count's behavior under the truncation of the StyleGAN generated samples on AFHQ and FFHQ datasets.
worse coverage of the underlying Gaussian mixture. The rest of our numerical results on mixtures of Gaussians are discussed in the Appendix.
Diversity Evaluation for Real Image Datasets. on the mentioned image datasets. Table contains the evaluated results. Also, since the IS is a combination of both diversity and quality factors, we propose the following coommationtthoretii decomposition of IS to IS quality and IS diveersi::
We measured the proposed and baseline scores
IS(X) == exp(I(x;))) = exp (()(H(())) exp())(())-()
where we call (Y)) IS IS-diversity and exp( H (+(x)) IS IS-quality. The decomposed Inception and KID scores are presented in the complete table of our numerical evaluations in the Appendix.
Our numerical results show that the RKE score consistently agrees with the majority of other diversity scores, and also for all GANs remains lower than the actual dataset's RKE. Therefore, due to the estimation guarantee in Theorem 6 our numerical results provide an information -tteorrtttt numerical proof for the empirical result in [15] that applies the birthday paradox to show GAN models cannot capture the full diversity of training data. In addition, the recent generative models StyleGAN-XL and ADM-G achieved the highest RKE, showing their diversity improvement over other generative model baselines.
Also, while the coverage and IS IS-diversity scores were able to differentiate between the CIFAR-IO- trained generative models, WGAN-WC had the lowest score for coverage and recall, despite the Inception score reporting as the most diverse case. Meanwhile, RKE ranked the absolute diversity of WGAN-WC to be similar to DCGAN while RRKE score shows that the modes captured by WGAN-WC are not common with that of the dataset. In the ImageNet experiments, RKE scores suggested that ContraGAN's samples are more diverse than SAGAN and SNGAN, while its coverage and recall scores were lower than those baselines. However, we note that ContraGAN reached a worse RRKE but its better RKE indicates that it captures a diverse set of modes that could have smaller intersection with the actual ImageNet modes. The above evaluation of absolute vs. relative diversity of generated samples of ContraGAN was not revealed by the other evaluation metrics.

---

Centra
Contra GAN
200
BigGAN256
Recall
0.6
Coveruge
0.6
SAGAN
RKE-MC ME--MC
150
SAGAN
0.4
0.4
100
Cantm GIN
BigGAN256
30-
SAGAN
0.0
5000
10000
15000
20000
5000
10000
15000
20000
5000
10000
15000
20000
Number samples
Number of samples
Number of samples
Figure 3: Comparing convergence of Recall, Coverage, and RKE scores on ImageNet dataset.
To assess the correlation between data diversity and RKE score, we repeated the dataset truncation experiment 12 for the RKE-MC measurement over AFHQ and FFHQ datasets. The numerical results in Figure 2 indicate significant correlation between the truncation factor and the evaluated RKE. Also, we observed that the empirical RKE-MC scores manage to converge to the underlying RKE-MC using relatively few samples. Figure 3 plots the evaluated RKE-MC, recall, and coverage scores under different sample sizes, which shows RKE-MC can be estimated well with Rs 2000 data.
Diversity Evaluation for text-to-image generative models. We used the proposed RKE and RRKE scores evaluate text-to-image generative models. In our experiments, we evaluated the state-of- the-art text-to-ine rext-to-image generative model GigaGAN and BK-SDM on the MS- MS-COCO dataset. The numerical results in Table that the GigaGAN model achieves lower RRKE comparison to BK-SDM. On the other hand, BK-SDM reaches higher RKE-based absolute mode diversity compared to GigaGAN.
Zero-shot evaluation on 30K images from MSCOCO validation set for text-to-image generative models. RKE-MC (Mode Count) denotes exp (RKE).
Table 2:
Method
IS
FID L
RRKE
RKE-MC f
GigaGAN
33.34
9.09
0.39
58.78
BK-SDM (Base)
33.79
15.76
0.44
73.05
7 Conclusion
In this work, we proposed a diversity evaluation method for generative models based on entropy measures in quantum information theory. The proposed matrix-based Rényi entropy scores were shown to correlate with the number of modes in a mixture distribution with sub- Sab-Gaussian components and can be estimated from empirical data with theoretical guarantees. Our numerical results suggest that while state-of-the-art generative models reach a lower entropy-based diversity score than training data, the recent GAN and diffusion model architectures such as StyleGAN-XL and ADM manage significantly improve the diversity factor over earlier geneartive models. A future direction for our work is to extend the diversity evaluation to non-GAN and non-diffusion models such variational autoencoders (VAEs) and flow-based models. Also, studying the effects of the pre- pre-trained Inception model on RKE and RRKE evaluations and comparing their robustness the choice of pre-trained models vs. the baseline scores studied in (88 will be an interesting topic for future exploration.
Acknowledgments
The work Farzan Farnia was partially supported by a grant from the Research Grants Council the Hong Kong Special Administrative Region, China, Project 14209920 and by CUHK Direct Research Grant [CUHK Project No. 4055164). The work of Cheuk Ting Li was partially supported by an ECS grant from the Research Grants Council of the Hong Kong Special Administrative Region, China [Project No.: CUHK 24205621]. The authors also thank the anonymous reviewers and metareviewer for their constructive feedback and suggestions.
10

---

References
[1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, In Advances in neural information processing systems, pages 2672-2680, 2014.
[2] Jascha Sohl- Sohh-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265 PMLR, 2015.
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[4] Tero Karras, Samuli Laine, and Timo Aila. A bayle-baedd generator architecture for generative adversarial networks. In CVPR, pages 4401-4410, 2019.
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840-6851, 2020.
[6] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2204.06/25, 2022.
[7] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local Nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6629-6640, Red Hook, NY, USA, 2017. Curran Associates Inc.
[8] Mikolaj Binkowski, Dougal Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference Learning Representations, 2018.
[9] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alee Radford, Xi Chen, and Xi Chen. Improved techniques for training GANs. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 29. Curran Associates, Inc., 2016.
[10] Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. NIPS' 18, page 5234-5243, Red Hook, NY, USA, 2018. Curran Associates Inc.
[11] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved Precision and Recall Metric for Assessing Generative Models. Curran Associates Inc., Red Hook, NY, USA, 2019.
[12] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. Proceedings of the 37th International Conference Machine Learning, ICML'20. JMLR.org. 2020.
[13] Gregg Jaeger. Quantum information. Springer, 2007.
[14] Luis Gonzalo Sanchez Giraldo, Murali Rao, and Jose C Principe. Measures of entropy from data using infinitely divisible kernels. IEEE Transactions on Information Theory, 61(1):535-556 2014.
[15] Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do GANs learn the distribution? some theory and empiries. In International Conference on Learning Representations, 2018.
[16] Ali Borji. Pros and cons of gan evaluation measures: New developments. Computer Vision and Image Understanding. 215:103329 2022.
[17] Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R. Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, pages
2017.
11

---

[18] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICMI'17, page 214-223. JMI.. 2017.
[19] L. N. Vaserstein. Markov processes over denumerable products of spaces, describing large systems of automata, Problemy Peredachi Informatsii, 3:64-72, 1969.
[20] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study. In Advances in Neural Information Processing Systems, volume 31, 2018.
[21] Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them. In 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6069-6078, 2020.
[22] Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. LR-GAN: Layered recursive generative adversarial networks for image generation. In International Conference on Learning Representations, 2017.
[23] Ahmed Alaa, Boris Van Breugel, Evgeny S. Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? Sample-level metrics for evaluating and auditing generative models. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 290-306. PMLR, 17-23 Jul 2022.
[24] Martin Müller-Lennert, Frédéric Dupuis, Oleg Szehr, Serge Fehr, and Marco Tomamichel. On quantum rényi entropies: A new generalization and some properties. Journal of Mathematical Physics, 54(12):122205 2013.
[25] Yuxin Dong, Ticliang Gong, Shujian Yu, and Chen Li. Optimal randomized approximations for matrix-based rényi's entropy. IEEE Transactions on Information Theory, 2023.
[26] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 5769-5777 Red Hook, NY, USA,
2017. Curran Associates Inc.
[27] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, 2009,
(28) Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge, CS 231N, 2015.
[29] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision - ECCV 2014, pages 740-755. Springer International Publishing, 2014.
[30] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[31] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for genera- tive adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396-4405, 2019.
[32] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L.. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[33] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS), 2020.
[34] Rewon Child. Very deep {vae}s generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.
[35] Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley, Least squares generative adversarial networks. In Proceedings of the IEEE International Conference Computer Vision (ICCV), Oct 2017.
12

---

[36] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In 4th International Conference on Learn- ing Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[37] Han Zhang, lan Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Proceedings of the 36th International Conference Machine Learning, volume 97, pages 7354-7363, 09-15 Jun 2019.
[38] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018.
[39] Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. In H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21357-21388 Curran Associates, Inc., 2020.
[40] Minguk Kang, Woohyeon Joseph Shim, Minsu Cho, and Jaesik Park. Rebooting ACGAN: Auxiliary classifier GANs with stable training. In A. Beygelzimer, Y. Dauphin, P. Liang, and
J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[41] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Proc. NeurlPS, 2021.
[42] Axel Sauer, Katja Schwarz. and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings, SIGGRAPH '22, New York, NY, USA, 2022. Association for Computing Machinery.
[43] Minguk Kang, Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10660 10695, June 2022.
[45] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8780-8794. Curran Associates, Inc., 2021.
[46] Bo-Kyeong Kim, Hyoung- BoyunggKKyu& Song. Thibault Castells, and Shinkook Choi. Bk-sdm: Architec- turally compressed stable diffusion for efficient text-to-image generation. ICML Workshop on Efficient Systems for Foundation Models (ES-FoMo), 2023.
[47] MinGuk Kang, Joonghyuk Shin, and Jaesik Park. StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI),
[48] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fr) "ccht inception distance. arXiv preprint arXiv:2203.0600 2022.
13


Reviews:
[{"review_id": "OF9KSVLx8P", "reviewer": "Reviewer_uN6a", "summary": "This paper uses matrix-based Renyi entropy and relative entropy to evaluate the diversity of generative models; in which the former measures absolute diversity (as it correlates with the number of modes in a mixture distribution) and the latter measures relative diversity (as it is in the same expression with the quantum Renyi relative entropy).  Authors use their two measures to evaluate the generative quality of a series of GAN and diffusion model architectures; and make some observations. ", "questions": "1. Can you explain the difference with respect to definition in [Sanchez Giraldo et al., 2014]? How is your efficient computation compared to [Dong et al., 2023]?\n2. See the second point of weaknesses\n3. In Theorem 4, is it necessary that two distributions have the same number of modes?\n4. Can you further explain the computational complexity of your measures compared to previous popular scores?", "limitations": "Authors did not discuss potential limitations and negative societal impacts. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The general idea to use entropy-based measures to evaluate diversity of generative models makes sense to me.\n2. It is good that authors demonstrated closed-form expressions for both measures in case of mixture Gaussians or sub-Gaussians. ", "weaknesses": "1. I found the definition of the empirical Renyi kernel entropy (RKE) which relies on a kernel Gram matrix K is exactly the same to the\ndefinition of matrix-based Renyi's alpha-order entropy functional in [Sanchez Giraldo et al., 2014], which also has fast implementation in [Dong et al., 2023].\n\nIn this sense, it is improper to use the expressions like \"we define ...\". etc. \n\n[Sanchez Giraldo et al., 2014] Giraldo, Luis Gonzalo Sanchez, Murali Rao, and Jose C. Principe. \"Measures of entropy from data using infinitely divisible kernels.\" IEEE Transactions on Information Theory 61.1 (2014): 535-548.\n\n[Dong et al., 2023] Dong, Yuxin, et al. \"Optimal Randomized Approximations for Matrix-based Rényi's Entropy.\" IEEE Transactions on Information Theory (2023).\n\n2. Some claims are not well explained. For example, according to Theorems 1 and 2, the RKE correlates with the number of modes k. However, it does not reflect that RKE is propositional to k? or why RKE can be precisely used to quantify the value of k? Similarly, why RRKE can be used to quantify the number of common modes according to Theorem 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer uN6a for his/her time and feedback. The following is our response to the reviewer’s comments and questions:\n\n**1-  Our work’s contributions and the mentioned references**\n\n**Re:** We would like to clarify that we do not claim we are the first to define the matrix-based Renyi entropy (general or kernel-based) which is a well-known tool in quantum information theory. We believe our main contribution is to apply the matrix-based Renyi entropy for the evaluation of generative models and further to provide theoretical interpretation for the Renyi entropy evaluation as a mode-based diversity score for multi-modal distributions. To the best of our knowledge, these contributions are novel and have not been discussed in previous works.\n\nRegarding the discussed papers in the review, we thank the reviewer for pointing out the reference [Sanchez Giraldo et al., 2014]. We note that the only common result between our work and [Sanchez Giraldo et al., 2014] is Corollary 1 on the analytical derivation of the empirical RKE score which we will clarify in the revision. However, we would like to highlight that all the other theoretical results in Theorems 1-7 are independent from [Sanchez Giraldo et al., 2014]’s findings, which includes\n\n**1-** Theorems 1-4 on the mode-based diversity interpretation of RKE and RRKE for mixtures of Gaussians and sub-Gaussians,\\\n**2-** Theorem 6 on the sample complexity guarantee for the RKE score,\\\n**3-** Theorem 7 on the nuclear-norm-based computation of empirical RRKE score,\n\nWe sincerely hope that Reviewer uN6a will consider the above contributions and also the shown application of the results to the evaluation of generative models in his/her assessment of our work. We will be happy to further discuss the reviewer’s concern on the novelty of our theoretical and numerical results.\n\n**2- Comparison between our proposed algorithms for computing the RKE score and [Dong et al., 2023]’s method**\n\n**Re:** In our numerical evaluation of the RKE score, we focused mostly on order-2 matrix-based Renyi entropy which reduces to the Frobenius norm of the kernel matrix as shown in Corollary 1. This algorithm will require $O(n^2 d)$ computation for $n$ samples of dimension $d$. In addition, Theorem 5 implies a randomized algorithm estimating the expected value using empirical samples which requires $O(nd)$ computation for $n$ pair of fresh empirical samples. On the other hand, [Dong et al., 2023]’s computation method applies to a general order-$\\alpha$ matrix-based Renyi entropy.\n\nThe following table shows the time (in seconds) taken by the three algorithms in performing the computation of order-2 Renyi entropy. The results in the table indicate that [Dong et al., 2023]’s method and the Frobenius norm-based approach result in similar time complexity, while the randomized algorithm based on empirical expected value can significantly reduce the computation time. \n\n| Algorithms | 1000 samples | 2000 samples | 3000 samples |\n|:--------------:|:---------:|:---------:|:---------:|\nFrobenius norm (Ours) | 9.12 | 36.35 | 81.85 |\nEmpirical expected value (Ours) | 2.19 | 3.25 | 5.10 |\nDong et al. (Hutch++ based) | 8.97 | 35.9 | 82.08 |\n\n**3- “why can RKE be used to quantify the value of k?”**\n\n**Re:** The reduction of RKE to the number of modes $k$ depends on the uniformity of samples’ distribution across the $k$ modes. Under a uniform distribution of samples across modes ($\\omega_i =\\frac{1}{k}$ in Theorem 1), the approximate RKE score in Line 174 for well-separable modes will be $$-\\log\\Bigl(\\sum_{i=1}^k \\omega_i^2 \\Bigr) = -\\log\\Bigl(k\\times \\frac{1}{k^2}\\Bigr) = \\log(k)$$ which is the entropy of a uniform $k$-ary distribution.  In contrast, under a non-uniform distribution of samples across the $k$ modes, the raw number of modes could become a suboptimal measure of diversity because some of the modes may have little frequency. In that case, our results suggest that the RKE score will approximately reduce to the order-$2$ Renyi entropy of the distribution across the $k$ modes, i.e. $-\\log\\bigl(\\sum_{i=1}^k \\omega_i^2\\bigr)$. This reduction generalizes the mode-based diversity to non-uniformly distributed samples across modes. We will explain the connection more clearly in the revised text.\n\n**4-  Theorem 4 and equal number of modes for distribution $P,Q$**\n\n**Re:** Theorem 4 will hold even if the input multimodal distributions $P,Q$ have different numbers of modes. A relatively simple generalization of the statement to the case with an unequal number of modes between $P,Q$ (with $k\\neq r$ modes) is to add $r-k$ zero-probability modes to distribution $P$ to match the number of modes between $P,Q$. The zero-probability modes do not affect the theorem’s statement and therefore address a case with an unequal number of modes.\n\n**5- Computational complexity of proposed scores vs. baselines**\n\n**Re:** For the RKE score, we follow the randomized algorithm of estimating the expected value in Theorem 5 that leads to $O(nd)$ computational complexity for $n$ samples of dimension $d$. For the RRKE score, we compute the nuclear norm of the cross-kernel matrix $K_{X,Y}$ through the SVD algorithm which requires $O(\\min(mn^2,m^2n)d)$ under $n$ samples for $X$ and $m$ samples for $Y$. \n\nRegarding the baselines, the KID score requires computing the Frobenius norm of the difference of kernel matrices that would cost $O(\\max (m^2,n^2) d)$. The FID score requires computing the Frobenius norm of the square root of the empirical covariance matrices which will be $O(\\max(n,m)d^3)$. Finally, the precision and recall scores run a k-nearest neighbor search for the empirical samples that need $O(nm\\log(m)kd)$ and $O(nmd\\log(n)kd)$ computational complexity. We note that a proper computational complexity comparison should take into account how many samples $m,n$ are needed to properly estimate the score for which our results suggest the Gaussian RKE and RRKE can converge faster than the baselines. \n"}, {"review_id": "LwTokk7OEa", "reviewer": "Reviewer_YzmH", "summary": "This paper introduces Renyi Entropy to measure the number of modes in generated samples. To this end, two metrics called empirical Renyi Kernel Entropy (RKE) and Relative Renyi Kernel Entropy (RRKE) are defined to estimate the Renyi Entropy of the eigenvalues of the data's kernel function. Some theoretical analyses are provided to guarantee the convergence of the proposed metrics. Experiments on synthesized data and benchmarks show that the proposed metrics can produce reasonable measurements.", "questions": "See questions in the weakness part above.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper focuses on an important topic, the measurement of generative models, and proposes a seemly promising idea, i.e. using Renyi Entropy to estimate the diversity of generated samples. The theoretical analysis makes the proposed matrics convincing and solid. The experimental results given by the proposed metrics seem reasonable.\n", "weaknesses": "My biggest concern is the applicability of the proposed metrics to more generative models, as this paper mainly reports results on GANs (as well as two diffusion models on ImageNet), but no results on VAEs (e.g. NVAE). It is well known that the samples generated by VAEs are more diverse than GANs but has some flaws like blurriness. Whether the proposed metrics can deal with VAEs remains unknown.\n\nAnother weakness of this paper is the absence of an explanation of concepts, hence the paper is not self-contained enough and somewhat hard to follow. For example, what is the meaning of interpreting the eigenvalues of the kernel matrix as probability? Why apply Renyi Entropy to these eigenvalues? What is relative diversity? Readers are difficult to understand the motivation and soundness of this paper without being told these concepts.\n\nMoreover, I suggest the authors discuss the relations and differences between RKE/ RRKE and other metrics (IS, FID, etc.). This will help readers to understand the benefits and position of the proposed metrics, as well as provide the necessity for proposing new metrics in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer YzmH for his/her time and constructive feedback and suggestions. In the following, we respond to the comments and questions raised in the review:\n\n**1- Application of the proposed scores to VAEs**\n\n**Re:** Based on the reviewer’s recommendation, we have estimated the proposed RKE and RRKE scores as well as the baseline scores for the suggested NVAE [1] and VDVAE [2] generative models. Consistent with Reviewer YzmH’s intuition, we observed that the RKE score of the trained VDVAE significantly improves upon the RKE scores of tested GAN generative models. On the other hand, the RRKE of the VAE models do not improve upon the GAN models, suggesting that the VAE models could produce more diverse results that have lower intersection with the modes of the data distribution. The evaluation scores for the VAE and GAN models can be found in Table 1 of the submitted rebuttal PDF, which we will include in the revised paper.\n\n[1] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS), 2020 \\\n[2] Rewon Child. Very deep VAEs generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations (ICLR), 2021.\n\n\n**2- Application of Renyi Entropy to the eigenvalues of kernel matrix**\n\n**Re:** We would like to note that the main contribution of our paper is to provide an operational interpretation of the matrix-based Renyi entropy for the evaluation of generative models. As suggested by Theorems 1-4, our theoretical interpretation reveals that for a multimodal distribution with well-separable sub-Gaussian components, the eigenvalues of the kernel matrix provide an estimation of the mode frequencies and the proposed RKE and RRKE scores will approximately be the standard (non-matrix) Renyi entropy for the distribution over the modes. We believe this score will result in a useful evaluation of standard generative models due to the presence of multiple modes in standard distributions of image and text data.\n\n**3- “What is relative diversity? “** \n\n**Re:** In the paper, we use the term “absolute diversity” to refer to the frequency of modes in the generative model, which in the special case of uniformly-distributed samples across modes means how many modes exist in the generated samples. On the other hand, we use the term “relative diversity” to represent what portion of modes existing in the training data have been captured by the generative model. We will make this point clear in the text.\n\n**4- “Differences between RKE/ RRKE and other metrics (IS, FID, etc.)”**\n\n**Re:** The main difference between the proposed RKE and the Inception score is the dynamic selection of modes used in the RKE score compared to the static modes used in the Inception score. Based on the definition of Inception score, the modes are statically chosen to be the 1000 labels of the ImageNet dataset. On the other hand, the RKE score allows a dynamic identification of the modes after observing the generated data, which will be more flexible in application to non-ImageNet data, e,g, CIFAR10, AFHQ, and FFHQ datasets discussed in the paper. We will include this discussion in the text."}, {"review_id": "c7lSlnJjfy", "reviewer": "Reviewer_5vQM", "summary": "The authors propose Renyi Kernel Entropy (RKE) as a measure of diversity for generative models. A relative diversity variant, relative Renyi kernel entropy (RRKE) is also defined. Estimation error bounds are derived and the measure is applied to both toy data, where it behaves as expected, and to state of the art image generative models.", "questions": "Suggestions:\n* The abstract states \"we show that the RKE score can output the number of modes of a mixture of sub-Gaussian components.\" It should be clarified that this is an estimate that refers to well-separated modes.\n\n* Figure 1 should display RKE-MC (mode count) as opposed to REK to make it consistent with Table 1.\n\nl223: It seems this should say one over square root n convergence.", "limitations": "Yes, although see the first suggestion under \"Questions.\"", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The proposed methodology is novel to the problem of estimating the diversity of generative models, a problem of interest to the NeurIPS audience.\n* The scores are clearly introduced and well-motivated by their ability to count the number of well-separated modes in a Gaussian mixture model, which is confirmed by the results in Figure 1.\n* The estimation error bounds are a great contribution.\n* Figure 3 shows great empirical convergence.\n\nOverall, the paper tackles a well-defined problem and offers an attractive solution in a detailed, convincing way.", "weaknesses": "* In the manuscript, it is unclear how matrix-based Renyi entropy is used in quantum information theory and how that relates to a notion of sample diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 5vQM for his/her time and constructive suggestions and feedback on our work. In the following, we respond to the comments and questions in the review:\n\n**1- Discussion on how matrix-based Renyi entropy is used in quantum information theory**\n\n**Re:** The application of quantum information measures is motivated through considering different states in the generated data, which could be interpreted as the modes of a multimodal distribution that are separable by a distance $\\gg\\sigma$ when we use a Gaussian kernel with bandwidth $\\sigma$ to define similarities between samples. In the revision, we will further explain the motivation behind applying quantum information measures and how we attempt to theoretically formalize the motivation by analyzing mixtures of well-separable sub-Gaussians.\n\n**2- “It should be clarified that this is an estimate that refers to well-separated modes.”**\n\n**Re:** As recommended by the reviewer, we will clarify the separability condition of the modes in the abstract and introduction. \n\n**3- RKE Mode Count in Figure 1 and Typo in Line 223**\n\n**Re:** We thank the reviewer for pointing out the typos in Figure 1 and Line 223. Figure 1 already reports RKE mode count ($\\exp(\\mathrm{RKE})$). We will correct the typos in the revision."}, {"review_id": "o5xlJ51IbE", "reviewer": "Reviewer_3kP8", "summary": "This paper investigates an information theoretic approach in evaluating the diversity of generative models by using Rényi kernel entropy (RKE) and relative Rényi kernel entropy (RRKE). The RKE measures the absolute diversity of a multi-modal distribution, while RRKE measures the relative diversity given the data distribution. They theoretically approve computational efficiency of RKE and RRKE estimators with convergence guarantee. The proposed evaluation metric is applied to various image generative models on CIFAR-10, tiny-ImageNet, and ImageNet-1K datasets, showing its effectiveness.", "questions": "1. Since there are various open-source multi-modal generative models (e.g., text-to-image diffusion models). How one can leverage RKE or RRKE in such cases? To be specific, evaluating the text-to-image models is a non-trivial problem (as the context made from text is infinitely large compared to class-conditional one). Do you have any intuition on how to better diversity of text-to-image models with RKE or RRKE?  \n2. Can RRKE be used throughout training to improve diversity? Instead of using it as evaluation, is it possible to use Relative Rényi kernel entropy as a regularizer to promote diversity in learning generative models?\n", "limitations": "Some limitations of the proposed evaluation method could be useful for the readers, for example, see the weakness part.  ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Evaluating the diversity of generative model is an important problem, yet many works resort to the FID or Inception Score due to its simplicity. The proposed method provides an alternative approach in metric for generative models with information theoretic principle. The proposed theorems are clearly written and the usage of Rényi divergence is novel. Also, it is nice to see the sample complexity (e.g., how many samples do we need to compute the metric). ", "weaknesses": "The usage of RKE and RRKE depends on various hyper parameters, such as the choice of kernel, where the author choose RBF kernel as default, thus the bandwidth hyperparameter must be chosen with validation step. Also, different order of Rényi entropies might result in different results, where they focus on order 2 and 1/2 for both RKE and RRKE. Generalizing to various orders or studying the effect of an order of Rényi entropy could enrich the paper.  \n\nIn practical side, while the RKE and RRKE aligns with other diversity metrics, the paper does not compare the superiority between metrics. For example, how RKE and RRKE is more aligned with human judgement? If there is such human ranking, comparing rank-correlation (e.g., Spearman’s rho or Kendall’s delta) can better show the effectiveness of RKE and RRKE.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 3kP8 for his/her time and constructive suggestions and feedback on our work. In the following, we respond to the comments and questions raised in the review:\n\n**1- Selecting the order of Renyi entropy scores in applications of the RKE evaluation**\n\n**Re:** In this work, we have focused on the order-2 RKE score that can be supported by a sample complexity bound (Theorem 6) as well as a fast computation method via a randomized approach for estimating the expected value in Theorem 5’s formulation. To address the reviewer’s comment, we evaluated the matrix-based Renyi entropy score of different orders and summarized the numerical results in Figure 1 of the submitted rebuttal PDF. As shown in this figure, order-2 Renyi entropy can successfully distinguish the diversity performance of BigGAN-2048 and SAGAN. An interesting future direction is to theoretically analyze the effect of changing the order of Renyi entropy in the diversity evaluation task.\n\n**2-Tuning the kernel bandwidth hyperparameter of the RKE/RRKE scores**\n\n**Re:** We would like to refer the reviewer to Figure 4 in the supplementary PDF which shows the effect of different bandwidth hyperparameters in the resulting RKE/RRKE scores. This figure indicates a consistent ranking of our attempted generative models across different bandwidth values. \n\n\n**3- RKE/RRKE vs. other diversity evaluation scores**\n\n**Re:** As discussed in the introduction, our primary motivation for proposing the RKE/RRKE-based evaluation of generative models is to develop an information theoretic approach to the assessment problem. While entropy measures are standard diversity scores in information theory, the challenging entropy estimation task in high-dimensional computer vision settings prevents their application to the evaluation of state-of-the-art generative models. In this work, we show that quantum information measures can address this challenge by estimating the entropy of major modes in an underlying multimodal distribution.\n\nFurthermore, we show that the proposed scores can be efficiently estimated from a limited number of training data which is the key to achieve lower statistical and computational costs in the evaluation process. Our truncation-based numerical results in the paper’s Figure 2 confirm the success of the proposed score in quantifying the diversity of generated samples. Also, in Figure 3 we demonstrate the efficient estimation procedure of the RKE score vs. standard diversity evaluation metrics. Finally, we note that performing a theoretical comparison between different diversity assessment criteria will be an interesting future direction to our work. \n\n**4- Diversity evaluation of text-to-image generative models**\n\n**Re:** Based on the reviewer’s recommendation, we used the proposed RKE and RRKE scores to evaluate text-to-image generative models which is similar to computing FID-30k for these models. In our experiments, we evaluated the state-of-the-art text-to-image generative model GigaGAN[1] and BK-SDM[2] on the COCO dataset. The numerical results of the experiments can be found in the rebuttal PDF’s Table 2, showing that the GigaGAN model achieves a higher diversity score than BK-SDM.\n\n\n[1] Kang, M., Zhu, J.Y., Zhang, R., Park, J., Shechtman, E., Paris, S., & Park, T. Scaling up GANs for Text-to-Image Synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023,\\\n[2] Bo-Kyeong Kim, and Hyoung-Kyu Song, and Thibault Castells, and Shinkook Choi, “BK-SDM: Architecturally Compressed Stable Diffusion for Efficient Text-to-Image Generation”,  ICML 2023.\n\n**5- Applying the RRKE score to regularize the training of generative models**\n\n**Re:** Since state-of-the-art generative models are typically trained by first-order optimization algorithms, the application of RRKE to regularize their training will be challenging. We note that the empirical RRKE is a non-differentiable map of the generative model due to the presence of nuclear norm in its formulation (shown in Theorem 7). Finding differentiable proxies of the RRKE score to regularize the training of generative models is definitely an interesting future direction, which we will discuss in the conclusion section."}]

Author rebuttal:
We thank the reviewers for their time and feedback. We have responded to the reviewers' comments and questions under every review textbox. Here, we attach the rebuttal PDF containing the tables and figures discussed in our responses.

Meta-review: The paper presents a novel approach to evaluate the diversity of generative models by using  Rényi entropy. The paper analyses the problem theoretically, show how it measures the number of components of a Gaussian mixture model, as well as experiment on real image data.

The task of evaluating generative models is a very important task, and I believe this paper adds a new insightful perspective on the problem and would be of interest to the community.
Decision: Accept (poster)