Title: An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative Models; Evaluation in Learning; Information Measures
Abstract: The evaluation of generative models has received significant attention in the machine learning community.  When applied to a multi-modal distribution which is common among image datasets, an intuitive evaluation criterion is the number of modes captured by the generative model. While several scores have been proposed to evaluate the quality and diversity of a model's generated data, the correspondence between existing scores and the number of modes in the distribution is unclear. In this work, we propose an information-theoretic diversity evaluation method for multi-modal underlying distributions. We utilize the R\'enyi Kernel Entropy (RKE) as an evaluation score based on quantum information theory to measure the number of modes in generated samples. To interpret the proposed evaluation method, we show that the RKE score can output the number of modes of a mixture of sub-Gaussian components. We also prove estimation error bounds for estimating the RKE score from limited data, suggesting a fast convergence of the empirical RKE score to the score for the underlying data distribution. Utilizing the RKE score, we conduct an extensive evaluation of state-of-the-art generative models over standard image datasets. The numerical results indicate that while the recent algorithms for training generative models manage to improve the mode-based diversity over the earlier architectures, they remain incapable of capturing the full diversity of real data. Our empirical results provide a ranking of widely-used generative models based on the RKE score of their generated samples.

An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions

Abstract

The evaluation of generative models has received significant attention in the machine learning community. When applied to a multi-modal distribution which is common among image datasets, an intuitive evaluation criterion is the number of modes captured by the generative model. While several scores have been proposed to evaluate the quality and diversity of a model's generated data, the correspondence between existing scores and the number of modes in the distribution is unclear. In this work, we propose an information-theoretic diversity evaluation method for multi-modal underlying distributions. We utilize the R´ enyi Kernel Entropy (RKE) as an evaluation score based on quantum information theory to measure the number of modes in generated samples. To interpret the proposed evaluation method, we show that the RKE score can output the number of modes of a mixture of subGaussian components. We also prove estimation error bounds for estimating the RKE score from limited data, suggesting a fast convergence of the empirical RKE score to the score for the underlying data distribution. Utilizing the RKE score, we conduct an extensive evaluation of state-of-the-art generative models over standard image datasets. The numerical results indicate that while the recent algorithms for training generative models manage to improve the mode-based diversity over the earlier architectures, they remain incapable of capturing the full diversity of real data. Our empirical results provide a ranking of widely-used generative models based on the RKE score of their generated samples 1 .

1 Introduction

Deep generative models trained by generative adversarial networks (GANs) [1] and diffusion models [2] have achieved impressive results in various unsupervised learning settings [3, 4, 5, 6]. Due to their success in generating image samples with high visual quality, the analysis of large-scale generative models has received great attention in the machine learning community. In particular, the evaluation of generative models has been extensively studied in the recent literature to understand the benefits and drawbacks of existing approaches to training generative models.

To address the evaluation task for generative models, multiple assessment scores have been proposed in the literature. The existing evaluation metrics can be divided into two general categories: 1)

Cheuk Ting Li

distance-based metrics including Fr´ echet Inception Distance (FID) [7] and Kernel Inception distance (KID) [8] scores, which measure a distance between the learned generative model and data distribution, 2) quality-based metrics including the Inception score [9], precision and recall scores [10, 11], and density and coverage scores [12], aiming to measure the quality and diversity of the generated samples based on the confidence and variety of labels assigned by a pre-trained neural net on ImageNet.

On the other hand, a different measure of diversity that is popular and standard in the information theory literature is entropy. The primary challenge with an entropy-based approach to the assessment of generative models is the statistical costs of entropy estimation in the typical high-dimensional spaces of image data, requiring a sample size exponentially growing with the dimension of data. Consequently, without further assumptions on the data distribution, it will be statistically and computationally infeasible to estimate the entropy value for high-dimensional image data.

In this work, we propose a novel information-theoretic approach for the diversity evaluation of generative models. Our proposed approach targets multi-modal data distributions comprised of several distinct modes, which is an applicable assumption to image datasets with a cluster-based structure due to their latent color and shape-based features. In this approach, we follow the entropy calculation in quantum information theory [13, 14] and utilize matrix-based R´ enyi entropy scores to evaluate the variety of samples produced by a generative model.

Considering the R´ enyi entropy in the Gaussian kernel space, we propose R´ enyi kernel entropy (RKE) and relative R´ enyi kernel entropy (RRKE) scores to measure the absolute and relative diversity of a multi-modal distribution with respect to the actual data distribution. We develop computationally efficient methods for estimating these entropy scores from empirical data with statistical convergence guarantees. We also prove that the RKE score will converge to standard differential entropy as the Gaussian kernel bandwidth approaches zero.

We provide an interpretation of the RKE score by deriving its closed-form expression for benchmark Gaussian mixture models (GMMs). In the GMM case, we show that the proposed RKE score reveals the number of Gaussian components in the underlying GMM, which motivates the method's application to general mixture distributions. We further extend this interpretation to mixture models with sub-Gaussian modes, e.g. modes with a bounded support set, to support the RKE score in more general settings. We discuss the numerical performance of the proposed score in synthetic mixture settings. Our numerical results demonstrate the fast convergence of the RKE score from a limited number of synthetic mixture data.

Next, we present the results of our numerical experiments on the evaluation of various state-of-the-art GAN and diffusion models using the RKE and RRKE metrics. Our numerical evaluation of the RKE score shows the lower diversity obtained by standard GAN models compared to real training data, which provides an information-theoretic numerical proof complementary to the birthday-paradox empirical proof for the same result in [15]. Furthermore, our empirical results suggest that the recent GANand diffusion model architectures improve the mode-based variety of generated data over earlier GAN formulations. We can summarize the main contributions of this work as follows:

Proposing an information theoretic approach to evaluate the diversity of generative models

Developing computationally efficient methods to compute R´ enyi kernel entropy scores

Providing theoretical and numerical support for the proposed evaluation methodology in the benchmark setting of Gaussian and sub-Gaussian mixture models

Diversity evaluation of standard generative models using the information-theoretic method

2 Related Work

The evaluation of GAN-based generative models has been studied by a large body of related works. As surveyed in [16], several evaluation methods have been developed in the literature. The Inception score (IS) [9] uses the output of a pre-trained Inception-net model as features and proposes a score summing up the entropy-based diversity of assigned labels' distribution and confidence score averaged over the conditional labels' distribution. The modified IS (m-IS) in [17] substitutes the KL-divergence term in IS with a cross entropy term, which helps m-IS capture diversity within images from a certain class. Unlike these works, our proposed approach bases on matrix-based entropy scores which capture the number of clusters in a multi-modal distribution.

Also, several distance-based evaluation metrics have been proposed in the deep learning literature. The Wasserstein Critic [18] attempts to approximate the Wasserstein distance between the real and generated samples. The Fr´ echet Inception Distance (FID) [7] measures a distance based on the embedding of the last layer of the pre-trained Inception-net, where it fits multivariate Gaussian models to real and generated data and calculates their Fr´ echet distance [19]. [20] conduct a comprehensive comparison of several standard GAN architectures based on IS and FID scores, discussing the similarities and differences of the GANs' performance. As another variant of FID, [21] suggest a bias-free estimation of FID using quasi-Monte Carlo integration. In another related work, [8] propose Kernel Inception Distance (KID) as the squared maximum mean discrepancy (MMD) between two distributions. Adversarial accuracy and divergence scores in [22] utilize two classifiers to compute the closeness of distributions of real and fake data conditioned on category labels from the two classifiers. Unlike the above metrics, our proposed relative R´ enyi entropy score focuses on the number of common modes between real and fake data, and hence reduces the statistical complexity of estimating the entropy.

The diversity vs. quality tradeoff of GANs' generated samples has also been studied in multiple related works. [15] examine the diversity of GANs' data through a birthday paradox-based approach and suggest that the support set size of GANs' data could be smaller than real training data. The precision and recall evaluation by [10] assigns a two-dimensional score where precision is defined as the portion of fake data that can be generated by real distribution while recall is defined as the portion of real data that can be generated by the generative model. The improved precision and recall in [11] further address the sensitivity disadvantages of these scores by estimating the density function via the k-nearest neighbour method. Also, the density and coverage scores [12] provide a more robust version of precision and recall metrics to outliers. Additionally, [23] proposed a 3-dimensional metric, that measures the fidelity, diversity and generalization of models. We note that our work offers a complementary approach to the diversity evaluation for GANs, and since it directly estimates the matrix-based entropy from data, it requires a comparatively smaller sample size for proper estimation.

3 Preliminaries

3.1 Kernel-based Feature Maps and Representation

Throughout the paper, we use X ∈ X to denote the data vector. Also, we denote the kernel feature map by ϕ : R t → R d which gives us the kernel function k : R t × R t → R as the inner product of the corresponding feature vectors: k ( x , y ) = 〈 ϕ ( x ) , ϕ ( y ) 〉 . Given n training data x 1 , . . . , x n we use Φ to denote the normalized kernel feature map-based data matrix, i.e.,

Then, the kernel matrix K ∈ R n × n whose ( i, j ) th entry will be 1 n k ( x i , x j ) will be identical to K = ΦΦ ⊤ . Observe that, by definition, K will be a positive semi-definite (PSD) matrix, possessing positive eigenvalues λ 1 , . . . , λ n where the non-zero eigenvalues are shared with the empirical kernel covariance matrix C = Φ ⊤ Φ . We call a kernel function normalized if k ( x , x ) = 1 for every x ∈ X . A standard example of a normalized kernel function is the Gaussian kernel with bandwidth parameter σ defined as:

For every normalized kernel function, the eigenvalues of kernel matrix K (and also empirical kernel covariance C ) will be non-negative and add up to 1 as the trace of K will be 1 . As a result, K 's eigenvalues can be interpreted as a probability sequence.

3.2 R´ enyi Entropy for PSD Matrices

A standard extension of the entropy concept to PSD matrices is the matrix-based R´ enyi entropy [14]. The R´ enyi entropy of order α > 0 for a PSD matrix A ∈ R d × d with eigenvalues λ 1 , . . . , λ d is defined as

where Tr denotes the trace operator. A commonly-used special case which we use throughout the paper is the R´ enyi entropy of order α = 2 defined as RE 2 ( A ) = log ( 1 / ∑ d i =1 λ 2 i ) .

Proposition 1. For every PSD A R , the following holds where denotes the Frobenius norm:

3.3 Relative R´ enyi Entropy

To measure the relative diversity of a matrix A ∈ R d × d with respect to another matrix B ∈ R d × d , one can use the sandwiched relative R´ enyi entropy of order α [24] defined as

A widely-used special case is the relative entropy of order α = 1 2 which is commonly called the Fidelity score in quantum information theory. The definition of the Fidelity score is

We note that the relative R´ enyi entropy of order α = 2 requires an invertible matrix B which may not hold in the applications to multi-modal distributions with rank-deficient kernel covariance matrices as discussed in the next sections.

4 A Diversity Metric for Multi-modal Distributions

4.1 Kernel-based R´ enyi Entropy Scores

Given the kernel matrix K computed using the data x 1 , . . . , x n of random vector X , the empirical R´ enyi kernel entropy (RKE) of the observed data can be defined as

which, as discussed in [14], is a diversity measure of the data. This diversity measurement approach will be interpreted and justified in the next subsection. Note that if the dimension d of the feature space is finite, since K = ΦΦ ⊤ and empirical covariance matrix ̂ C = Φ ⊤ Φ share the same eigenvalues, we have

Therefore, we can see that the empirical R´ enyi kernel entropy ̂ RKE α ( X ) is an estimate of the following quantity about the underlying distribution P X where X is sampled from, which we call the R´ enyi kernel entropy of P X :

Here C X denotes the kernel covariance matrix of distribution P X defined as

Similarly, we can use the kernel-based relative R´ enyi entropy as a measure of joint diversity between distributions P X , P Y of random vectors X , Y . Here, for random vectors X , Y distributed according to P X , P Y , we define the relative R´ enyi kernel entropy (RRKE α ) score as the orderα relative kernel entropy between their kernel covariance matrices C X , C Y :

In order to estimate the above relative entropy score, we can use the empirical RRKE score between the empirical kernel covariance matrices for samples x 1 , . . . , x n from P X and samples y 1 , . . . , y m from P Y :

In the rest of this section, we show how the kernel-based R´ enyi entropy score evaluated under a Gaussian kernel can relate to the number of modes of multi-modal distributions with sub-Gaussian components, and subsequently how the relative R´ enyi entropy score counts the number of joint modes between two multi-modal distributions with sub-Gaussian modes.

∈ d ∥· ∥ F .

4.2 RKE as a Measure of the Number of Modes

We consider a multi-modal underlying distribution P X consisting of k distinct modes. Here, our goal is to show that the order-2 RKE score under the Gaussian kernel can count the number of present modes in the distribution. To this end, we analyze the order-2 R´ enyi kernel entropy score of the kernel covariance matrix C X of mixtures of Gaussian and sub-Gaussian components and theoretically show that the RKE score reduces to the number of well-separated modes.

First, we derive the closed-form expression of the order-2 R´ enyi kernel entropy of a Gaussian mixture model under the Gaussian kernel. Here, we use the following notation to denote a k -component Gaussian mixture model where the i th component has frequency ω i , mean vector µ i and Covariance matrix Σ i : P GMM( ω , µ , Σ) = ∑ k i =1 ω i N ( µ i , Σ i ) . Also, for a positive definite matrix A ∈ R d × d , we use ∥ · ∥ A to denote the A -norm defined as ∥ x ∥ A = √ x ⊤ A x .

Theorem 1. Suppose that the distribution of X is given by the Gaussian mixture model P GMM( ω , µ , Σ) . Then, order-2 R´ enyi kernel score under the Gaussian kernel with bandwidth σ , denoted by G σ , is

where A i,j is defined as follows given the d × d identity matrix I :

Proof. We defer the proof to the Appendix.

Corollary 1. Suppose X ∼ GMM( ω , µ , Σ) follows from a Gaussian mixture model with isotropic covariance matrices Σ i = σ 2 i I . Defining the coefficients a i,j := 1 + 2( σ 2 i + σ 2 j ) σ 2 , the RKE score will be

Theorem 1 and Corollary 1 show that if σ ≫ max i ∥ Σ i ∥ sp , i.e. the kernel bandwidth dominates the spectral norm (maximum eigenvalue) of the component-wise covariance matrices, then A i,j ≈ I and a i,j ≈ 1 and the RKE score will approximately be

The next theorem shows the above approximation generalizes to mixtures of sub-Gaussian components, e.g. modes with bounded support sets, and also provides an approximation error bound.

Theorem 2. Suppose that X ∈ R d has a mixture distribution P SGMM = ∑ k i =1 ω i P i where the k th component occurs with probability ω k and has a sub-Gaussian distribution with parameter σ i , i.e. its moment-generating function (MGF) M P i satisfies the following for every vector β ∈ R d given the mean vector µ i for P i : E P i [ exp ( β ⊤ ( X -µ i ) )] ≤ exp ( ∥ β ∥ 2 2 σ 2 i / 2 ) . Then, the following approximation error bound holds where we define α 2 i := 1 + 2 σ 2 i /σ 2 :

Proof. We defer the proof to the Appendix.

More generally, we show that the RKE under a Gaussian kernel with a small bandwidth provides an estimate of the differential R´ enyi entropy, which connects the approach with a smoothed differential entropy estimator in high-dimensional settings.

Theorem 3. Suppose that the underlying distribution of X has a continuous probability density function P X . The order2 R´ enyi kernel score under Gaussian kernel with bandwidth σ for this underlying distribution satisfies

where the right hand side is the order2 R´ enyi differential entropy of P X .

Proof. We defer the proof to the Appendix.

4.3 RRKE as the Number of Common Modes

In order to measure the joint mode-based diversity, we propose the order1 2 relative R´ enyi entropy score. We note that our choice of order 1 2 is due to the existing inverse covariance matrix term in the relative entropies of orders greater than 1 , which is not applicable to rank deficient matrices expected in the case of well-separated multi-modal distributions. Furthermore, the order 1 2 relative entropy, well-known as the fidelity score, is a commonly-used and well-analyzed relative entropy case in quantum information theory.

To interpret the application of the order1 2 relative entropy, we show the following theorem discussing how the negative RRKE score could approximate the joint diversity between two input multimodal distributions with sub-Gaussian components.

Theorem 4. Suppose that X , Y ∈ R d are random vectors with mixture distributions P MM 1 = ∑ k i =1 ω i P i and P MM 2 = ∑ k i =1 η i Q i , respectively, where ω i , η i denote the frequency of the i th component. We also assume that P i and Q i have mean vectors µ i and ζ i respectively and are both σ i -sub-Gaussian. Then, the following approximation error bound holds for order1 2 relative R´ enyi entropy where we define α 2 i = 1 + 2 σ 2 i /σ 2 :

Proof. We defer the proof to the Appendix.

The above theorem shows that if the i th mode of mixture distribution P mm 1 and the j th mode of mixture distribution P mm 2 are sufficiently close, then they add nearly √ ω i η j to the RRKE score.

5 Estimation Procedure and Guarantees

As discussed earlier, the RKE and RRKE scores for a proper kernel function provide measures of the absolute and relative mode-based diversity of multi-modal distributions. Here, we propose kernelbased estimators for these entropy scores. Our estimators suggest computationally and statistically feasible ways for approximating the entropy measures by exploiting the connections between the kernel similarity values and the R´ enyi entropy scores. In addition, we provide non-asymptotic estimation error bounds for the proposed estimators to analyze their sample complexity. Regarding the RKE score, the following results connect this score with the kernel function k ( x , x ′ ) .

Theorem 5. Given a random vector X distributed as P X , the order-2 RKE score is the result of the following equation, where X , X ′ are IID draws of P X :

Corollary 2. For samples x 1 , . . . , x n and K XX = [ 1 n k ( x i , x j )] n × n being the normalized kernel matrix for the observed samples, the empirical order-2 Kernel entropy is given by

Proof. We defer the proof to the Appendix. We note that the result of Corollary 2 for the empirical RKE score has already been shown and discussed in [14].

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

RKE-MC: 1.11, RRKE: 2.04

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

LSGAN

RKE-MC: 3.30, RRKE: 0.98

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

WGAN-WC

RKE-MC: 8.80, RRKE: 0.06

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

WGAN-GP

RKE-MC: 8.93, RRKE: 0.02

1

.

5

1

.

0

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

RKE-MC: 3.97, RRKE: 1.83

1

.

5

1

.

0

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

RKE-MC: 2.10, RRKE: 2.34

1

.

5

1

.

0

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

RKE-MC: 26.49, RRKE: 0.06

1

.

5

1

.

0

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

RKE-MC: 25.17, RRKE: 0.04

-

-

-

-

-

-

-

-

-

-

-

-

As the above results suggest, we can use the normalized Frobenius norm of the kernel matrix to estimate the RKE score. Therefore, the computation of order-2 RKE only requires the Frobenius norm of the kernel matrix, which can be computed in O ( n 2 ) complexity. We note that for a general orderα R´ enyi entropy, one can apply the randomized algorithm in [25] for the computation of the orderα RKE score. Our next result bounds the estimation error for the empirical RKE score.

Theorem 6. Consider a normalized kernel function satisfying k ( x , x ) = 1 for every x ∈ X . Then, for every δ > 0 the following bound will hold with probability at least 1 -δ :

Proof.

As implied by the above results, the order-2 R´ enyi kernel entropy can be efficiently estimated from training data and the probability of an ϵ -large error will exponentially diminish with the sample size n . Next, we discuss the computation approach for the order1 2 RRKE score. The following result reveals the kernel-based representation of this score in the empirical case.

Theorem 7. Consider empirical samples x 1 , . . . , x n drawn from P X and y 1 , . . . , y m drawn from P Y . Then, the following identity holds for their empirical order1 2 RRKE score:

where K XY = [ 1 √ nm k ( x i , y j ) ] n × m denotes the normalized cross kernel matrix and ∥ · ∥ ∗ denotes the nuclear norm, i.e. the sum of a matrix's singular values.

Proof.

As implied by the above theorem, the RRKE score can be computed using the singular value decomposition (SVD) for finding the singular values of K XY . Note that the application of SVD requires O (min { m 2 n, mn 2 } ) computations given n, m samples from P X and P Y , respectively. Therefore, computing the RRKE score could be more expensive than the RKE score, since the nuclear norm is more costly to compute than the Frobenius norm.

6 Numerical Results

We tested the performance of the proposed entropy-based diversity evaluation approach on several combinations of standard GAN architectures and datasets. Specifically, we used the synthetic 8component and 25-component Gaussian mixture datasets in [26] and the following image datasets:

CIFAR-10 [27], Tiny-ImageNet [28], MS-COCO [29], AFHQ [30], FFHQ [31] and ImageNet [32]. We evaluated the performance of the following list of widely-used VAE, GAN and diffusion model architectures: NVAE [33], Very Deep VAE (VDVAE) [34], Vanilla GAN [1], LSGAN [35], Wassesrtein GAN with weight clipping (WGAN-WC) [18], Wassesrtein GAN with gradient penalty (WGAN-GP) [26], DCGAN [36], Self-Attention GAN (SAGAN) [37], Spectrally-Normalized GAN (SNGAN) [38], ContraGAN [39], ReACGAN [40], BigGAN [3], StyleGAN3 [41], StyleGAN-XL [42], GigaGAN [43], LDM [44] ADM-G [45] and BK-SDM [46]. To have a fair evaluation of the models, we downloaded the trained generative models from the StudioGAN repository [47].

In our evaluation of generative models, we compared the performance of order-2 R´ enyi Kernel Entropy Mode Count (RKE-MC), defined as exp(RKE 2 ( X )) , and order1 2 Relative R´ enyi Kernel Entropy (RRKE) with the following standard baselines widely used in the evaluation of generative models: Inception Score (IS) [9], Fr´ echet Inception Distance (FID) [7], Kernel Inception Distance (KID) [8], precision and recall [11], density and coverage [12].

To compute the RKE and RRKE scores for the Gaussian mixture cases, we measured the scores based on the output of the trained generator. For image datasets, we followed the standard approach in the literature and evaluated the scores for the representation of the generator's output characterized by an Inception-net-V3 model pre-trained on ImageNet. We note that the Inception-net-based evaluation methodology is consistent with the baseline methods. Also, to select the bandwidth parameter σ for the Gaussian kernel in the RKE and RRKE scores, we performed cross-validation and chose the smallest bandwidth σ for which the reported score's standard deviation across 5,000 validation samples is below 0.01. Note that if the kernel bandwidth becomes overly small, the RKE score will grow almost logarithmically with the number of samples, and the standard deviation of its exponential, i.e. RKE mode count (RKE-MC), will increase almost linearly with the sample size and thus suffer from a large variance across disjoint sample sets. We provide a more detailed discussion of the bandwidth parameter's selection and the resulting variance in the Appendix.

Diversity Evaluation for Synthetic Mixture Datasets . We measured the RKE-MC and RRKE scores for the 8 and 25 component Gaussian mixture datasets in [26]. Figure 1 shows the real samples in blue and GANs' generated samples in red. As the evaluated scores suggest, the RKE-MC scores managed to count the number of captured modes and the RRKE relative distance increased under a

y

=

0

.

0

y

=

0

.

3

y

=

0

.

6

y

=

1

.

0

(a) AFHQ generated samples with different

ψ

.

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Truncation

y

0

10

20

30

40

50

RKE-MC

AFHQ Dataset

FFHQ Dataset

y

=

0

.

2

y

=

0

.

4

y

=

0

.

6

y

=

0

.

8

y

=

1

.

0

(c) FFHQ generated samples with different ψ .

worse coverage of the underlying Gaussian mixture. The rest of our numerical results on mixtures of Gaussians are discussed in the Appendix.

Diversity Evaluation for Real Image Datasets . We measured the proposed and baseline scores on the mentioned image datasets. Table 1 contains the evaluated results. Also, since the IS is a combination of both diversity and quality factors, we propose the following information-theoretic decomposition of IS to IS-quality and IS-diversity:

where we call exp( H ( ˆ Y )) IS-diversity and exp( -H ( ˆ Y | X )) IS-quality. The decomposed Inception and KID scores are presented in the complete table of our numerical evaluations in the Appendix.

Our numerical results show that the RKE score consistently agrees with the majority of other diversity scores, and also for all GANs remains lower than the actual dataset's RKE. Therefore, due to the estimation guarantee in Theorem 6, our numerical results provide an information-theoretic numerical proof for the empirical result in [15] that applies the birthday paradox to show GAN models cannot capture the full diversity of training data. In addition, the recent generative models StyleGAN-XL and ADM-G achieved the highest RKE, showing their diversity improvement over other generative model baselines.

Also, while the coverage and IS-diversity scores were able to differentiate between the CIFAR-10trained generative models, WGAN-WC had the lowest score for coverage and recall, despite the Inception score reporting it as the most diverse case. Meanwhile, RKE ranked the absolute diversity of WGAN-WC to be similar to DCGAN while RRKE score shows that the modes captured by WGAN-WC are not common with that of the dataset. In the ImageNet experiments, RKE scores suggested that ContraGAN's samples are more diverse than SAGAN and SNGAN, while its coverage and recall scores were lower than those baselines. However, we note that ContraGAN reached a worse RRKE but its better RKE indicates that it captures a diverse set of modes that could have a smaller intersection with the actual ImageNet modes. The above evaluation of absolute vs. relative diversity of generated samples of ContraGAN was not revealed by the other evaluation metrics.

0

5000

10000

15000

20000

Number of samples

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Recall

Contra GAN

BigGAN256

SAGAN

0

5000

10000

15000

20000

Number of samples

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Coverage

Contra GAN

BigGAN256

SAGAN

0

5000

10000

15000

20000

Number of samples

50

100

150

200

250

RKE-MC

Contra GAN

BigGAN256

SAGAN

To assess the correlation between data diversity and RKE score, we repeated the dataset truncation experiment of [11, 12] for the RKE-MC measurement over AFHQ and FFHQ datasets. The numerical results in Figure 2 indicate a significant correlation between the truncation factor and the evaluated RKE. Also, we observed that the empirical RKE-MC scores manage to converge to the underlying RKE-MC using relatively few samples. Figure 3 plots the evaluated RKE-MC, recall, and coverage scores under different sample sizes, which shows RKE-MC can be estimated well with ≈ 2000 data.

Diversity Evaluation for text-to-image generative models. We used the proposed RKE and RRKE scores to evaluate text-to-image generative models. In our experiments, we evaluated the state-ofthe-art text-to-image generative model GigaGAN and BK-SDM on the MS-COCO dataset. The numerical results in Table 2 indicate that the GigaGAN model achieves lower RRKE in comparison to BK-SDM. On the other hand, BK-SDM reaches higher RKE-based absolute mode diversity compared to GigaGAN.

7 Conclusion

In this work, we proposed a diversity evaluation method for generative models based on entropy measures in quantum information theory. The proposed matrix-based R´ enyi entropy scores were shown to correlate with the number of modes in a mixture distribution with sub-Gaussian components and can be estimated from empirical data with theoretical guarantees. Our numerical results suggest that while state-of-the-art generative models reach a lower entropy-based diversity score than training data, the recent GAN and diffusion model architectures such as StyleGAN-XL and ADM manage to significantly improve the diversity factor over earlier geneartive models. A future direction for our work is to extend the diversity evaluation to non-GAN and non-diffusion models such as variational autoencoders (VAEs) and flow-based models. Also, studying the effects of the pre-trained Inception model on RKE and RRKE evaluations and comparing their robustness to the choice of pre-trained models vs. the baseline scores studied in [48] will be an interesting topic for future exploration.

Acknowledgments

The work of Farzan Farnia was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China, Project 14209920 and by a CUHK Direct Research Grant [CUHK Project No. 4055164]. The work of Cheuk Ting Li was partially supported by an ECS grant from the Research Grants Council of the Hong Kong Special Administrative Region, China [Project No.: CUHK 24205621]. The authors also thank the anonymous reviewers and metareviewer for their constructive feedback and suggestions.

References

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems , pages 2672-2680, 2014.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2019.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR , pages 4401-4410, 2019.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS , 33:6840-6851, 2020.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 6629-6640, Red Hook, NY, USA, 2017. Curran Associates Inc.

Mikołaj Bi´ nkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMDGANs. In International Conference on Learning Representations , 2018.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training GANs. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. NIPS'18, page 5234-5243, Red Hook, NY, USA, 2018. Curran Associates Inc.

Tuomas Kynk¨ a¨ anniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved Precision and Recall Metric for Assessing Generative Models . Curran Associates Inc., Red Hook, NY, USA, 2019.

Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In Proceedings of the 37th International Conference on Machine Learning , ICML'20. JMLR.org, 2020.

Gregg Jaeger. Quantum information . Springer, 2007.

Luis Gonzalo Sanchez Giraldo, Murali Rao, and Jose C Principe. Measures of entropy from data using infinitely divisible kernels. IEEE Transactions on Information Theory , 61(1):535-548, 2014.

Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do GANs learn the distribution? some theory and empirics. In International Conference on Learning Representations , 2018.

Ali Borji. Pros and cons of gan evaluation measures: New developments. Computer Vision and Image Understanding , 215:103329, 2022.

Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R. Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , volume 1, pages 4941-4949, 2017.

Martin Arjovsky, Soumith Chintala, and L´ eon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 214-223. JMLR.org, 2017.

L. N. Vaserstein. Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii , 5:64-72, 1969.

Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In Advances in Neural Information Processing Systems , volume 31, 2018.

Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6069-6078, 2020.

Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. LR-GAN: Layered recursive generative adversarial networks for image generation. In International Conference on Learning Representations , 2017.

Ahmed Alaa, Boris Van Breugel, Evgeny S. Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? Sample-level metrics for evaluating and auditing generative models. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 290-306. PMLR, 17-23 Jul 2022.

Martin M¨ uller-Lennert, Fr´ ed´ eric Dupuis, Oleg Szehr, Serge Fehr, and Marco Tomamichel. On quantum r´ enyi entropies: A new generalization and some properties. Journal of Mathematical Physics , 54(12):122203, 2013.

Yuxin Dong, Tieliang Gong, Shujian Yu, and Chen Li. Optimal randomized approximations for matrix-based r´ enyi's entropy. IEEE Transactions on Information Theory , 2023.

Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 5769-5779, Red Hook, NY, USA, 2017. Curran Associates Inc.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, 2009.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge, CS 231N, 2015.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision - ECCV 2014 , pages 740-755. Springer International Publishing, 2014.

Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4396-4405, 2019.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09 , 2009.

Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS) , 2020.

Rewon Child. Very deep { vae } s generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations , 2021.

Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , Oct 2017.

Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016.

Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Proceedings of the 36th International Conference on Machine Learning , volume 97, pages 7354-7363, 09-15 Jun 2019.

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations , 2018.

Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 21357-21369. Curran Associates, Inc., 2020.

Minguk Kang, Woohyeon Joseph Shim, Minsu Cho, and Jaesik Park. Rebooting ACGAN: Auxiliary classifier GANs with stable training. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Tero Karras, Miika Aittala, Samuli Laine, Erik H¨ ark¨ onen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Proc. NeurIPS , 2021.

Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings , SIGGRAPH '22, New York, NY, USA, 2022. Association for Computing Machinery.

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 8780-8794. Curran Associates, Inc., 2021.

Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: Architecturally compressed stable diffusion for efficient text-to-image generation. ICML Workshop on Efficient Systems for Foundation Models (ES-FoMo) , 2023.

MinGuk Kang, Joonghyuk Shin, and Jaesik Park. StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2023.

Tuomas Kynk¨ a¨ anniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fr \ 'echet inception distance. arXiv preprint arXiv:2203.06026 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates an information theoretic approach in evaluating the diversity of generative models by using Rényi kernel entropy (RKE) and relative Rényi kernel entropy (RRKE). The RKE measures the absolute diversity of a multi-modal distribution, while RRKE measures the relative diversity given the data distribution. They theoretically approve computational efficiency of RKE and RRKE estimators with convergence guarantee. The proposed evaluation metric is applied to various image generative models on CIFAR-10, tiny-ImageNet, and ImageNet-1K datasets, showing its effectiveness.\n\nSTRENGTHS:\nEvaluating the diversity of generative model is an important problem, yet many works resort to the FID or Inception Score due to its simplicity. The proposed method provides an alternative approach in metric for generative models with information theoretic principle. The proposed theorems are clearly written and the usage of Rényi divergence is novel. Also, it is nice to see the sample complexity (e.g., how many samples do we need to compute the metric).\n\nWEAKNESSES:\nThe usage of RKE and RRKE depends on various hyper parameters, such as the choice of kernel, where the author choose RBF kernel as default, thus the bandwidth hyperparameter must be chosen with validation step. Also, different order of Rényi entropies might result in different results, where they focus on order 2 and 1/2 for both RKE and RRKE. Generalizing to various orders or studying the effect of an order of Rényi entropy could enrich the paper.  \n\nIn practical side, while the RKE and RRKE aligns with other diversity metrics, the paper does not compare the superiority between metrics. For example, how RKE and RRKE is more aligned with human judgement? If there is such human ranking, comparing rank-correlation (e.g., Spearman’s rho or Kendall’s delta) can better show the effectiveness of RKE and RRKE.\n\nQUESTIONS:\n1. Since there are various open-source multi-modal generative models (e.g., text-to-image diffusion models). How one can leverage RKE or RRKE in such cases? To be specific, evaluating the text-to-image models is a non-trivial problem (as the context made from text is infinitely large compared to class-conditional one). Do you have any intuition on how to better diversity of text-to-image models with RKE or RRKE?  \n2. Can RRKE be used throughout training to improve diversity? Instead of using it as evaluation, is it possible to use Relative Rényi kernel entropy as a regularizer to promote diversity in learning generative models?\n\nLIMITATIONS:\nSome limitations of the proposed evaluation method could be useful for the readers, for example, see the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors propose Renyi Kernel Entropy (RKE) as a measure of diversity for generative models. A relative diversity variant, relative Renyi kernel entropy (RRKE) is also defined. Estimation error bounds are derived and the measure is applied to both toy data, where it behaves as expected, and to state of the art image generative models.

STRENGTHS:
* The proposed methodology is novel to the problem of estimating the diversity of generative models, a problem of interest to the NeurIPS audience.
* The scores are clearly introduced and well-motivated by their ability to count the number of well-separated modes in a Gaussian mixture model, which is confirmed by the results in Figure 1.
* The estimation error bounds are a great contribution.
* Figure 3 shows great empirical convergence.

Overall, the paper tackles a well-defined problem and offers an attractive solution in a detailed, convincing way.

WEAKNESSES:
* In the manuscript, it is unclear how matrix-based Renyi entropy is used in quantum information theory and how that relates to a notion of sample diversity.

QUESTIONS:
Suggestions:
* The abstract states "we show that the RKE score can output the number of modes of a mixture of sub-Gaussian components." It should be clarified that this is an estimate that refers to well-separated modes.

* Figure 1 should display RKE-MC (mode count) as opposed to REK to make it consistent with Table 1.

l223: It seems this should say one over square root n convergence.

LIMITATIONS:
Yes, although see the first suggestion under "Questions."
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Renyi Entropy to measure the number of modes in generated samples. To this end, two metrics called empirical Renyi Kernel Entropy (RKE) and Relative Renyi Kernel Entropy (RRKE) are defined to estimate the Renyi Entropy of the eigenvalues of the data's kernel function. Some theoretical analyses are provided to guarantee the convergence of the proposed metrics. Experiments on synthesized data and benchmarks show that the proposed metrics can produce reasonable measurements.\n\nSTRENGTHS:\nThis paper focuses on an important topic, the measurement of generative models, and proposes a seemly promising idea, i.e. using Renyi Entropy to estimate the diversity of generated samples. The theoretical analysis makes the proposed matrics convincing and solid. The experimental results given by the proposed metrics seem reasonable.\n\nWEAKNESSES:\nMy biggest concern is the applicability of the proposed metrics to more generative models, as this paper mainly reports results on GANs (as well as two diffusion models on ImageNet), but no results on VAEs (e.g. NVAE). It is well known that the samples generated by VAEs are more diverse than GANs but has some flaws like blurriness. Whether the proposed metrics can deal with VAEs remains unknown.\n\nAnother weakness of this paper is the absence of an explanation of concepts, hence the paper is not self-contained enough and somewhat hard to follow. For example, what is the meaning of interpreting the eigenvalues of the kernel matrix as probability? Why apply Renyi Entropy to these eigenvalues? What is relative diversity? Readers are difficult to understand the motivation and soundness of this paper without being told these concepts.\n\nMoreover, I suggest the authors discuss the relations and differences between RKE/ RRKE and other metrics (IS, FID, etc.). This will help readers to understand the benefits and position of the proposed metrics, as well as provide the necessity for proposing new metrics in this paper.\n\nQUESTIONS:\nSee questions in the weakness part above.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses matrix-based Renyi entropy and relative entropy to evaluate the diversity of generative models; in which the former measures absolute diversity (as it correlates with the number of modes in a mixture distribution) and the latter measures relative diversity (as it is in the same expression with the quantum Renyi relative entropy).  Authors use their two measures to evaluate the generative quality of a series of GAN and diffusion model architectures; and make some observations.\n\nSTRENGTHS:\n1. The general idea to use entropy-based measures to evaluate diversity of generative models makes sense to me.\n2. It is good that authors demonstrated closed-form expressions for both measures in case of mixture Gaussians or sub-Gaussians.\n\nWEAKNESSES:\n1. I found the definition of the empirical Renyi kernel entropy (RKE) which relies on a kernel Gram matrix K is exactly the same to the\ndefinition of matrix-based Renyi's alpha-order entropy functional in [Sanchez Giraldo et al., 2014], which also has fast implementation in [Dong et al., 2023].\n\nIn this sense, it is improper to use the expressions like \"we define ...\". etc. \n\n[Sanchez Giraldo et al., 2014] Giraldo, Luis Gonzalo Sanchez, Murali Rao, and Jose C. Principe. \"Measures of entropy from data using infinitely divisible kernels.\" IEEE Transactions on Information Theory 61.1 (2014): 535-548.\n\n[Dong et al., 2023] Dong, Yuxin, et al. \"Optimal Randomized Approximations for Matrix-based Rényi's Entropy.\" IEEE Transactions on Information Theory (2023).\n\n2. Some claims are not well explained. For example, according to Theorems 1 and 2, the RKE correlates with the number of modes k. However, it does not reflect that RKE is propositional to k? or why RKE can be precisely used to quantify the value of k? Similarly, why RRKE can be used to quantify the number of common modes according to Theorem 4.\n\nQUESTIONS:\n1. Can you explain the difference with respect to definition in [Sanchez Giraldo et al., 2014]? How is your efficient computation compared to [Dong et al., 2023]?\n2. See the second point of weaknesses\n3. In Theorem 4, is it necessary that two distributions have the same number of modes?\n4. Can you further explain the computational complexity of your measures compared to previous popular scores?\n\nLIMITATIONS:\nAuthors did not discuss potential limitations and negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}