SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The above theorem shows that if the  $i$ th mode of mixture distribution  $P_{\mathrm{mm}_1}$  and the  $j$ th mode of mixture distribution  $P_{\mathrm{mm}_2}$  are sufficiently close, then they add nearly  $\sqrt{\omega_i \eta_j}$  to the RRKE score.
5 Estimation Procedure and Guarantees
As discussed earlier, the RKE and RRKE scores for a proper kernel function provide measures of the absolute and relative mode-based diversity of multi-modal distributions. Here, we propose kernel-based estimators for these entropy scores. Our estimators suggest computationally and statistically feasible ways for approximating the entropy measures by exploiting the connections between the kernel similarity values and the Rényi entropy scores. In addition, we provide non-asymptotic estimation error bounds for the proposed estimators to analyze their sample complexity. Regarding the RKE score, the following results connect this score with the kernel function  $k(\mathbf{x},\mathbf{x}^{\prime})$ .
Theorem 5. Given a random vector  $\mathbf{X}$  distributed as  $P_{X}$ , the order-2 RKE score is the result of the following equation, where  $\mathbf{X},\mathbf{X}^{\prime}$  are IID draws of  $P_{X}$ :
$$
\mathrm {R K E} _ {2} (\mathbf {X}) = - \log \Bigl (\mathbb {E} _ {X, X ^ {\prime} \stackrel {\mathrm {i i d}} {\sim} P _ {X}} \left[ k ^ {2} (\mathbf {X}, \mathbf {X} ^ {\prime}) \right] \Bigr).
$$
Corollary 2. For samples  $\mathbf{x}_1, \ldots, \mathbf{x}_n$  and  $K_{XX} = \left[\frac{1}{n} k(\mathbf{x}_i, \mathbf{x}_j)\right]_{n \times n}$  being the normalized kernel matrix for the observed samples, the empirical order-2 Kernel entropy is given by
$$
\widehat {\mathrm {R K E}} _ {2} (\mathbf {X}) = - \log \left(\left\| K _ {X X} \right\| _ {F} ^ {2}\right),
$$
Proof. We defer the proof to the Appendix. We note that the result of Corollary 2 for the empirical RKE score has already been shown and discussed in [14].  $\square$
Figure 1: GANs' generated samples for Gaussian mixtures and the RKE-MC  $\uparrow$  and RRKE  $\downarrow$  scores.
As the above results suggest, we can use the normalized Frobenius norm of the kernel matrix to estimate the RKE score. Therefore, the computation of order-2 RKE only requires the Frobenius norm of the kernel matrix, which can be computed in  $O(n^{2})$  complexity. We note that for a general order- $\alpha$  Rényi entropy, one can apply the randomized algorithm in [25] for the computation of the order- $\alpha$  RKE score. Our next result bounds the estimation error for the empirical RKE score.
Theorem 6. Consider a normalized kernel function satisfying  $k(\mathbf{x},\mathbf{x}) = 1$  for every  $\mathbf{x} \in \mathcal{X}$ . Then, for every  $\delta >0$  the following bound will hold with probability at least  $1 - \delta$ :
$$
\left| \exp \bigl (- \widehat {\mathrm {R K E} _ {2}} (\mathbf {X}) \bigr) - \exp \bigl (- \mathrm {R K E} _ {2} (\mathbf {X}) \bigr) \right| \leq O \Big (\sqrt {\frac {\log n}{n}} \Big)
$$
As implied by the above results, the order-2 Rényi kernel entropy can be efficiently estimated from training data and the probability of an  $\epsilon$ -large error will exponentially diminish with the sample size  $n$ . Next, we discuss the computation approach for the order- $\frac{1}{2}$  RRKE score. The following result reveals the kernel-based representation of this score in the empirical case.
Theorem 7. Consider empirical samples  $\mathbf{x}_1, \ldots, \mathbf{x}_n$  drawn from  $P_X$  and  $\mathbf{y}_1, \ldots, \mathbf{y}_m$  drawn from  $P_Y$ . Then, the following identity holds for their empirical order- $\frac{1}{2}$  RRKE score:
$$
\widehat {\mathrm {R R K E}} _ {\frac {1}{2}} (\mathbf {X}, \mathbf {Y}) = - \log \left(\left\| K _ {X Y} \right\| _ {*} ^ {2}\right)
$$
where  $K_{XY} = \left[\frac{1}{\sqrt{nm}} k(\mathbf{x}_i,\mathbf{y}_j)\right]_{n\times m}$  denotes the normalized cross kernel matrix and  $\| \cdot \|_*$  denotes the nuclear norm, i.e. the sum of a matrix's singular values.
As implied by the above theorem, the RRKE score can be computed using the singular value decomposition (SVD) for finding the singular values of  $K_{XY}$ . Note that the application of SVD requires  $O(\min\{m^2 n, mn^2\})$  computations given  $n, m$  samples from  $P_X$  and  $P_Y$ , respectively. Therefore, computing the RRKE score could be more expensive than the RKE score, since the nuclear norm is more costly to compute than the Frobenius norm.
6 Numerical Results
We tested the performance of the proposed entropy-based diversity evaluation approach on several combinations of standard GAN architectures and datasets. Specifically, we used the synthetic 8-component and 25-component Gaussian mixture datasets in [26] and the following image datasets:
Table 1: Evaluated scores for three image datasets. RKE-MC (Mode Count) denotes exp(RKE).
Method	IS ↑	FID ↓	Precision ↑	Recall ↑	Density ↑	Coverage ↑	RRKE ↓	RKE-MC ↑
Dataset	11.57	-	-	-	-	-	-	39.58
NVAE	5.85	51.67	0.36	0.50	0.28	0.60	2.01	17.65
VDVAE	10.51	37.51	0.34	0.78	0.23	0.21	1.81	32.49
DCGAN	5.75	54.30	0.59	0.25	0.49	0.23	0.98	10.19
WGAN-WC	2.59	157.26	0.36	0.00	0.18	0.03	2.09	10.64
WGAN-GP	7.51	21.66	0.62	0.56	0.57	0.51	0.74	19.07
SAGAN	8.62	10.17	0.68	0.62	0.73	0.73	0.65	24.46
SNGAN	8.81	9.23	0.70	0.62	0.77	0.74	0.62	25.83
ContraGAN	9.69	4.02	0.75	0.62	0.99	0.86	0.52	29.80
Tiny-ImageNet	33.99	-	-	-	-	-	-	155.86
SAGAN	8.21	46.98	0.55	0.49	0.44	0.27	1.42	25.68
SNGAN	8.12	48.96	0.55	0.46	0.40	0.26	1.46	27.18
BigGAN	11.57	27.34	0.60	0.58	0.53	0.43	1.23	39.61
ContraGAN	13.79	21.36	0.54	0.54	0.54	0.45	1.26	56.94
ImageNet	357.35	-	-	-	-	-	-	1823.52
SAGAN-256	29.67	44.66	0.57	0.58	0.42	0.35	2.34	105.57
SNGAN-256	31.92	35.75	0.54	0.64	0.41	0.38	2.22	115.62
ContraGAN-256	24.91	34.79	0.67	0.51	0.64	0.33	2.54	152.89
BigGAN-256	28.33	33.48	0.58	0.61	0.49	0.37	2.28	106.07
ReACGAN-256	52.53	15.65	0.74	0.42	0.79	0.41	2.15	119.76
BigGAN-2048	96.42	4.49	0.71	0.58	0.80	0.65	1.83	606.18
StyleGAN-XL	204.73	1.94	0.77	0.61	0.67	0.81	1.50	1375.17
LDM-4-G	242.62	3.60	0.86	0.60	0.69	0.78	1.56	1321.24
ADM-G	188.70	3.86	0.82	0.64	0.66	0.82	1.47	1407.75
CIFAR-10 [27], Tiny-ImageNet [28], MS-COCO [29], AFHQ [30], FFHQ [31] and ImageNet [32]. We evaluated the performance of the following list of widely-used VAE, GAN and diffusion model architectures: NVAE [33], Very Deep VAE (VDVAE) [34], Vanilla GAN [1], LSGAN [35], Wassestein GAN with weight clipping (WGAN-WC) [18], Wassestein GAN with gradient penalty (WGAN-GP) [26], DCGAN [36], Self-Attention GAN (SAGAN) [37], Spectrally-Normalized GAN (SNGAN) [38], ContraGAN [39], ReACGAN [40], BigGAN [3], StyleGAN3 [41], StyleGAN-XL [42], GigaGAN [43], LDM [44] ADM-G [45] and BK-SDM [46]. To have a fair evaluation of the models, we downloaded the trained generative models from the StudioGAN repository [47].
In our evaluation of generative models, we compared the performance of order-2 Rényi Kernel Entropy Mode Count (RKE-MC), defined as  $\exp (\mathrm{RKE}_2(\mathbf{X}))$ , and order- $\frac{1}{2}$  Relative Rényi Kernel Entropy (RRKE) with the following standard baselines widely used in the evaluation of generative models: Inception Score (IS) [9], Fréchet Inception Distance (FID) [7], Kernel Inception Distance (KID) [8], precision and recall [11], density and coverage [12].
To compute the RKE and RRKE scores for the Gaussian mixture cases, we measured the scores based on the output of the trained generator. For image datasets, we followed the standard approach in the literature and evaluated the scores for the representation of the generator's output characterized by an Inception-net-V3 model pre-trained on ImageNet. We note that the Inception-net-based evaluation methodology is consistent with the baseline methods. Also, to select the bandwidth parameter  $\sigma$  for the Gaussian kernel in the RKE and RRKE scores, we performed cross-validation and chose the smallest bandwidth  $\sigma$  for which the reported score's standard deviation across 5,000 validation samples is below 0.01. Note that if the kernel bandwidth becomes overly small, the RKE score will grow almost logarithmically with the number of samples, and the standard deviation of its exponential, i.e. RKE mode count (RKE-MC), will increase almost linearly with the sample size and thus suffer from a large variance across disjoint sample sets. We provide a more detailed discussion of the bandwidth parameter's selection and the resulting variance in the Appendix.
Diversity Evaluation for Synthetic Mixture Datasets. We measured the RKE-MC and RRKE scores for the 8 and 25 component Gaussian mixture datasets in [26]. Figure 1 shows the real samples in blue and GANs' generated samples in red. As the evaluated scores suggest, the RKE-MC scores managed to count the number of captured modes and the RRKE relative distance increased under a
(a) AFHQ generated samples with different  $\psi$
(b) RKE Mode Count for StyleGAN generated samples on AFHQ and FFHQ varying with truncation factor  $\psi$
$\psi = 0.2$
Figure 2: RKE mode count's behavior under the truncation of the StyleGAN generated samples on AFHQ and FFHQ datasets.
$\psi = 0.4$
(c) FFHQ generated samples with different  $\psi$
$\psi = 0.6$
$\psi = 0.8$
$\psi = 1.0$
worse coverage of the underlying Gaussian mixture. The rest of our numerical results on mixtures of Gaussians are discussed in the Appendix.
Diversity Evaluation for Real Image Datasets. We measured the proposed and baseline scores on the mentioned image datasets. Table 1 contains the evaluated results. Also, since the IS is a combination of both diversity and quality factors, we propose the following information-theoretic decomposition of IS to IS-quality and IS-diversity:
$$
\operatorname {I S} (X) := \exp \bigl (I (X; \hat {Y}) \bigr) = \exp \bigl (H (\hat {Y}) \bigr) \exp \bigl (- H (\hat {Y} | X) \bigr),
$$
where we call  $\exp (H(\hat{Y}))$  IS-diversity and  $\exp (-H(\hat{Y} |X))$  IS-quality. The decomposed Inception and KID scores are presented in the complete table of our numerical evaluations in the Appendix.
Our numerical results show that the RKE score consistently agrees with the majority of other diversity scores, and also for all GANs remains lower than the actual dataset's RKE. Therefore, due to the estimation guarantee in Theorem 6, our numerical results provide an information-theoretic numerical proof for the empirical result in [15] that applies the birthday paradox to show GAN models cannot capture the full diversity of training data. In addition, the recent generative models StyleGAN-XL and ADM-G achieved the highest RKE, showing their diversity improvement over other generative model baselines.
Also, while the coverage and IS-diversity scores were able to differentiate between the CIFAR-10-trained generative models, WGAN-WC had the lowest score for coverage and recall, despite the Inception score reporting it as the most diverse case. Meanwhile, RKE ranked the absolute diversity of WGAN-WC to be similar to DCGAN while RRKE score shows that the modes captured by WGAN-WC are not common with that of the dataset. In the ImageNet experiments, RKE scores suggested that ContraGAN's samples are more diverse than SAGAN and SNGAN, while its coverage and recall scores were lower than those baselines. However, we note that ContraGAN reached a worse RRKE but its better RKE indicates that it captures a diverse set of modes that could have a smaller intersection with the actual ImageNet modes. The above evaluation of absolute vs. relative diversity of generated samples of ContraGAN was not revealed by the other evaluation metrics.
Figure 3: Comparing convergence of Recall, Coverage, and RKE scores on ImageNet dataset.
To assess the correlation between data diversity and RKE score, we repeated the dataset truncation experiment of [11, 12] for the RKE-MC measurement over AFHQ and FFHQ datasets. The numerical results in Figure 2 indicate a significant correlation between the truncation factor and the evaluated RKE. Also, we observed that the empirical RKE-MC scores manage to converge to the underlying RKE-MC using relatively few samples. Figure 3 plots the evaluated RKE-MC, recall, and coverage scores under different sample sizes, which shows RKE-MC can be estimated well with  $\approx 2000$  data.
Diversity Evaluation for text-to-image generative models. We used the proposed RKE and RRKE scores to evaluate text-to-image generative models. In our experiments, we evaluated the state-of-the-art text-to-image generative model GigaGAN and BK-SDM on the MS-COCO dataset. The numerical results in Table 2 indicate that the GigaGAN model achieves lower RRKE in comparison to BK-SDM. On the other hand, BK-SDM reaches higher RKE-based absolute mode diversity compared to GigaGAN.
Table 2: Zero-shot evaluation on 30K images from MSCOCO validation set for text-to-image generative models. RKE-MC (Mode Count) denotes exp(RKE).
Method	IS ↑	FID ↓	RRKE ↓	RKE-MC ↑
GigaGAN	33.34	9.09	0.39	58.78
BK-SDM (Base)	33.79	15.76	0.44	73.05
In this work, we proposed a diversity evaluation method for generative models based on entropy measures in quantum information theory. The proposed matrix-based Rényi entropy scores were shown to correlate with the number of modes in a mixture distribution with sub-Gaussian components and can be estimated from empirical data with theoretical guarantees. Our numerical results suggest that while state-of-the-art generative models reach a lower entropy-based diversity score than training data, the recent GAN and diffusion model architectures such as StyleGAN-XL and ADM manage to significantly improve the diversity factor over earlier geneartive models. A future direction for our work is to extend the diversity evaluation to non-GAN and non-diffusion models such as variational autoencoders (VAEs) and flow-based models. Also, studying the effects of the pre-trained Inception model on RKE and RRKE evaluations and comparing their robustness to the choice of pre-trained models vs. the baseline scores studied in [48] will be an interesting topic for future exploration.
The work of Farzan Farnia was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China, Project 14209920 and by a CUHK Direct Research Grant [CUHK Project No. 4055164]. The work of Cheuk Ting Li was partially supported by an ECS grant from the Research Grants Council of the Hong Kong Special Administrative Region, China [Project No.: CUHK 24205621]. The authors also thank the anonymous reviewers and metareviewer for their constructive feedback and suggestions.
[36] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[37] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 7354–7363, 09–15 Jun 2019.
[38] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[39] Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21357-21369. Curran Associates, Inc., 2020.
[40] Minguk Kang, Woohyeon Joseph Shim, Minsu Cho, and Jaesik Park. Rebooting ACGAN: Auxiliary classifier GANs with stable training. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[41] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Proc. NeurIPS, 2021.
[42] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings, SIGGRAPH '22, New York, NY, USA, 2022. Association for Computing Machinery.
[43] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[45] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8780-8794. Curran Associates, Inc., 2021.
[46] Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: Architecturally compressed stable diffusion for efficient text-to-image generation. ICML Workshop on Efficient Systems for Foundation Models (ES-FoMo), 2023.
[47] MinGuk Kang, Joonghyuk Shin, and Jaesik Park. StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023.
[48] Tuomas Kynkänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fr\`echet inception distance. arXiv preprint arXiv:2203.06026, 2022.