Towards a Scalable Reference-Free Evaluation of Generative Models
Authors: Azim Ospanov
Jingwei Zhang
Mohammad Jalali
Xuenan Cao
Andrej Bogdanov
Farzan Farnia
Venue: NeurIPS 2024 poster (2024)
Keywords: Evaluation of Generative Models
Scalable Algorithms
Kernel Methods
Random Fourier Features
Primary area: generative_models

Abstract: While standard evaluation scores for generative models are mostly reference-based, a reference-dependent assessment of generative models could be generally difficult due to the unavailability of applicable reference datasets. Recently, the reference-free entropy scores, VENDI and RKE, have been proposed to evaluate the diversity of generated data. However, estimating these scores from data leads to significant computational costs for large-scale generative models. In this work, we leverage the random Fourier features framework to reduce the metrics' complexity and propose the *Fourier-based Kernel Entropy Approximation (FKEA)* method. We utilize FKEA's approximated eigenspectrum of the kernel matrix to efficiently estimate the mentioned entropy scores. Furthermore, we show the application of FKEA's proxy eigenvectors to reveal the method's identified modes in evaluating the diversity of produced samples. We provide a stochastic implementation of the FKEA assessment algorithm with a complexity $O(n)$ linearly growing with sample size $n$. We extensively evaluate FKEA's numerical performance in application to standard image, text, and video datasets. Our empirical results indicate the method's scalability and interpretability applied to large-scale generative models.  The codebase is available at [https://github.com/aziksh-ospanov/FKEA](https://github.com/aziksh-ospanov/FKEA).

Towards a Scalable Reference-Free Evaluation of Generative Models
Azim Ospanov"
Jingwei Zhang" jwzhang228cse. cuhk. edu. hk
aospanov98cse. cuhk. edu hk
Mohammad Jalali"
Xuenan Cao
Andrej Bogdanov i
mjalali240cse. cuhk. edu hk
xuenancao@cuhk. edu hk
abogdano@uottaww..co ca
Farzan Farnia"
farnia@cse. cuhk. edu. .kk
Abstract
While standard evaluation scores for generative models are mostly eeference-basedd reference- -erreededde assessment of generative models could be generally difficult due to the unavailability of applicable reference datasets. Recently, the reference- free entropy scores, VENDI [ and RKE (2]. have been proposed evaluate the diversity of generated data. However, estimating these scores from data leads to significant computational costs for large-scale generative models. this work, we leverage the random Fourier features framework reduce the computational price and propose the Fourier-based Kernel Entropy Approximation (FKEA) method. We utilize FKEA's approximated eigenspectrum of the kernel matrix to efficiently estimate the mentioned entropy scores. Furthermore, we show the application FKEA's proxy eigenvectors reveal the method's 's identified modes in evaluating the diversity of produced samples. We provide stochastic implementation of the FKEA assessment algorithm with complexity (nn) linearly growing with sample size n. We extensively evaluate FKEA's numerical performance in application standard image, text, and video datasets. Our empirical results indicate the method's scalability and interpretability applied to large-scale generative models. The codebase available attttps://e/ thub. com/ faikkshoae www.kkhhhopp//EEEAA
Introduction
quantitative comparison of generative models requires evaluation metrics to measure the quality and diversity of the models' produced data. Since the introduction of variational autoencoders (VAFs) 3. generative adversarial networks (GANs) 4 and diffusion models (5] that led to impressive empirical results in the last decade, several evaluation scores have been proposed to assess generative models learned by different training methods and architectures. Due the key role of evaluation criteria in comparing generative models, they have been extensively studied in the literature.
While various statistical methods have been applied measure the fidelity and variety of generative model's 's produced data, the standard scores commonly perform a reference-basee evaluation of generative models, i.c., they quantify the characteristies of generated samples in comparison to reference distribution. The reference distribution is usually chosen to be either the distribution of
DDepartment Computer Science Engineering, The Chinese University Hong Kong. Hong Kong Depaarmennt of Cultural and Religious Studies, The Chinese University of Hong Kong. Hong Kong School Electrical Engineering and Computer Science, University of Outawa, Canada
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"regular elephant, detailed"
Identified Clusters
Recall-0.013
Conerage
FKEA-VENDI-461
"[color] elephant, detailed
Identified Clusters
FID=117.35 Recall=0.015
CoveragenB.05G
PREA-VENDEEIL.S
FKEA-RKE-2.16
Figure 1: Reference- based vs, eeference-free scores on two datasets of Stable Diffusion XL generated elephant images. FID, Recall, and Coverage scores (colored orange) reference-based, whereas VENDI and RKE scores (colored blue) are reference-free. Inception. V3 is used as the backbone embedding. Reference-based metrics use 'Indian elephant' samples in ImageNet reference data.
samples in the test data partition or comprehensive dataset containing a significant fraction of real-world sample types, e.g. ImageNet ( for evaluating image-based generative models.
To provide well-known examples of reference-dependent metrics, note that the distance scores, Fréchet Inception Distance (FID) 7 and Kernel Inception Distance (KID) & are explicitly reference- based, measuring the distance between the generative and reference distributions. Similarly, the standard quality/diversity score pairs, Precision/Recall [ 10 and Density/Coverage (T., perform the evaluation in comparison to reference dataset. Even the seemingly reference-free Inception Score (IS) [12] can be viewed implicitly reference-based, since quantifies the variety and fidelity of data based on the labels and confidence scores assigned by an ImageNet pre-trained neural net, where ImageNet implicitly plays the role of the reference dataset. The reference-basee nature of these evaluation scores is desired in many instances including standard image- based generative models, where either sufficiently large test set or a comprehensive reference dataset such as ImageNet is available for the reference-bart evaluation.
On the other hand, reference-baseee assessment of generative models may always be feasible, because the selection of a reference distribution may be challenging in a general learning scenario. For example, prompt-based generative models where the data are created in response user's input text prompts, the generated data could follow an priori unknown distribution depending on the specific distribution of the user's input prompts. Figure | shows one such example where we compare reference-bar based diversity scores of regular and colored elephant image samples generated by Stable Diffusion XL. B While the diversity of the colored images looks significantly higher to the human eye, the evaluated reference- based FID, Recall, and Coverage metrics do not suggest higher diversity. As this example suggests, proper reference- based evaluation of every user's generated data would require distinct reference dataset, which may not be available to the user during the assessment time. Moreover, finding a comprehensive text or video dataset to choose as the reference set would be more difficult compared to image data, because the higher length of text and video samples could significantly contribute to their varicty, requiring an inefficiently large reference set to cover all text or video sample types.
The discussed challenging scenarios of conducting a reference-based evaluation highlight the need for reference-fie free assessment methods that remain functional in the absence of reference dataset. Recently, entropy-base based diversity evaluation scores, the VENDI metric family [[] 14 and RKE score (2]. have been proposed to address the need for reference-free assessment metrics. These scores calculate the entropy of the eigenvalues
kernel similarity matrix for the generated data. Based on the theoretical results in [2], the evaluation process of these scores can be interpreted as an unsupervised identification of the generative model's produced sample clusters, followed by the entropy calculation for the frequencies of the detected clusters. In Figure | observe that the

---

reference-free free VENDI and RKE scores grow when the generated samples are colored, which is due to the increase in the quantity of identified clusters in the colored case.
While the VENDI and RKE. entropy scores provide reference-free assessments of generative models, estimating these scores from generated data could incur significant computational costs. In this work, we show that computing the precise RKE and VENDI scores would require at least {2(n²) and (nn23373 computations for sample size n, respectively. While the randomized projection methods in I can reduce the computational costs to (n²)) for general VENDL, score, the quadratic growth would be barrier the method's application to large n values. Although the computational expenses could be reduced by limiting the sample size n, an insufficient sample size would lead to significant eтror in estimating the entropy scores. As an example on the ImageNet dataset, Figure 7 in the Appendix shows the adverse effects of limiting the sample size on the quality of clusters used in the calculation of the VENDI scores.
To overcome the challenges of computing the scores, we leverage the random Fourier features (RFFs) framework [16] and develop scalable entropy-based evaluation method that can be efficiently applied to large sample sizes. Our proposed method, Fourier-based Kernel Entropy Approximation (FKEA), designed to approximate the kernel covariance matrix using the RFFs drawn from the Fourier transform-inverse of target shift-invariant kernel. We prove that using a Fourier feature size r = O Iog FKEA computes the eigenspace of the kernel matrix within an e-bounded error. Furthermore, we demonstrate the application of the eigenvectors of the FKEA's proxy kernel matrix for identifying the sample clusters used in the reference-fre evaluation of entropic diversity.
Finally, we present numerical results of the entropy-based evaluation of standard generative models using the baseline eigendecomposition and our proposed FKEA methods. In our experiments, the baseline spectral decomposition algorithm could not efficiently scale to sample sizes above a few ten thousand. On the other hand, our stochastic implementation of the FKEA method could scalably apply to large sample sizes. Utilizing the standard embeddings of image, text, and video data, we tested the FKEA assessment while computing the sample clusters and their frequencies in application to large-scale datasets and generative models. Here is a summary of our work's main contributions:
Characterizing the computational complexity of the kernel entropy scores of generative models, Developing the Fourier-based FKEA method to approximate the kemel covariance eigenspace and entropy of generated data,
Proving guarantees FKEA's required size of random Fourier features indicating complexity logarithmically growing with the dataset size,
Providing numerical results on FKEA's reference-free assessment of large-scale image,text, video- based datasets and generative models.
2 Related Work
Evaluation of deep generative models. The assessment of generative models has been widely studied in the literature. The existing scores either quantify distance between the distributions of real and generated data, as in FID (7] and KID (8) scores, or attempt to measure the quality and diversity of the trained generative models, including the Inception Score [12], quality/diversity metric pairs Precision/Recall 99, 10 and Density/Coverage 11.. The mentioned scores are reference-
based, while in this work we focus reference- metrics. Also, note that the evaluation of memorization and novelty has received great attention, and several scores including the authenticity score (17), the feature likelihood divergence [88, and the rarity score [19] have been proposed to quantify the generalizability and novelty of generated samples. Note that the evaluation of novelty and generalization is, by nature, reference-based based. On the other hand, our study focuses on the diversity of data which can be evaluated in reference-free way as discussed in 2
Role of embedding in quantitative evaluation. Following the discussion in 20), we utilize D2 21 image embeddings in most of our image experiments, as [20 ss results indicate D2 can yield scores more aligned with the human notion of diversity. As noted in (22. it is possible to utilize other non-ImageNet feature spaces such as CLIP 23 and SWAV 24 as opposed to Inception 25 to
'This computation complexity is the minimum known achievable cost for multiplying n x matrices which we prove to lower-bound the complexity of computing matrix-based entropy scores.
3

---

further improve metrics such as FID. In this work, we mainly focus on DinoV2 feature space, while we note that other feature spaces are also compatible with entropy- based diversity evaluation.
Diversity assessment for text-based models. To quantify the diversity of text data, the n-gram- methods are commonly used in the literature. A well-known metric is the BLEU score (26), which is based on the geometric average of n-gram precision scores times the Brevity Penalty. To adapt BLEU score measure text diversity, [27] proposes the Self-BLEU score, calculating the average BLEU score of various generated samples. To further isolate and measure diversity, N-Gram Diversity scores 29 3D were proposed and defined by a ratio between the number of unique n n-grams and overall 28
number of n-grams in the text. Other prominent metrics include Homogenization (ROUGE-L) 31
FBD 33 and Compression Ratios (33.
Kernel PCA, Spectral Cluttering, and Random Fourier Features. Kernel PCA 14 is a well- studied method of dimensionality reduction that utilizes the eigendecomposition of the kernel matrix, similar to the kernel- based diversity evaluation methods in 2 The related papers 35
36 study the connections between kernel PCA and spectral clustering. Also, the analysis of random Fourier features [16] for performing scalable kernel PCA has been studied in [37 38 (39 40 41 We note that while the mentioned works characterize the complexity of estimating the eigenvectors, our analysis focuses on the complexity of computing the kernel matrix's ' eigenvalues via Fourier features, as we primarily seek quantify the diversity of generated data using the kernel matrix's 's eigenvalues.
3 Preliminaries
Consider a generative model 9 generating random samples X1,..., X, E IRd following the model's probability distribution Pg. In our analysis, we assume the generated samples are independently drawn from Pg- Note that in VAEs B and GANs [4), the generative model g is deterministic function G JR' + R mapping an r-dimensional latent random vector ~ Pz from known distribution Pz to G(Z) distributed according to Pg. On the other hand, in diffusion models, G represents iterative random process that generates sample from Pç. The goal of sample-based diversity evaluation of generative model G is to quantify the variety of its generated data X1 Xx.
3.1 Kernel Function, Kernel Covariance Matrix, and Matrix-based Rényi Entropy
Following standard definitions, k Rd x Idd + R is called a kernel function if for every integer n E N and set of inputs Xi,..., . X,, E Rdd the kernel similarity matrix K [xxx, x,) nxn is positive semi-definite. We call a kernel function normalized for every input x we have k(x, x) = 1. A well-known example of a normalized kernel function is the Gaussian kernel (Gaussinn( with
bandwidth parameter a² defined
X
x11
(Gaussian (2)) (x, == exp(
20
For every kernel function k, there exists a feature map d : Rd + Rmm such that k(x, x') = (ø(x), d(x)) is the inner product of the m-dimennional feature maps o(x) and ø(x'). Given a kernel
k with feature map o, we define the kernel covariance matrix Cx E Rmmxm of distribution Px as
Cx = Ex~Px (()))())) = | px())((x))( dx
:
The above matrix Cx is positive semi-definite with non-negative values. Furthermore, assuming normalized kernel k; it can be seen that the eigenvalues of Cx will add up (i.e., it has unit trace TT(C')) = 1), providing probability model. Therefore, one can consider the entropy of Cx 's eigenvalues as a quantification of the diversity of distribution Px based on the kernel similarity score k. Here, we review the general family of Rényi entropy used to define VENDI and RKE scores.
Definition 1. For a positive semi-definite matrix Cx E Rmxx with eigenvalues A y An the order-o Rényi entropy Ha Ha(Cx) (Co)foorr a > is defined as
1
m
(Cx) = 1 log
a

---

To estimate the entropy scores from finite empirical samples X1.... Xn, we consider the empirical kernel covariance matrix Cx defined as Cx == Ei= (xxx) oxx:) This matrix provides an empirical estimation of the population kernel covariance matrix Cx.
It can be seen that the m X m empirical matrix Öx and normalized kernel matrix Н [xx;; [[)] nxn share the same non-zero eigenvalues. Therefore, to compute the matrix-based entropy of the empirical covariance matrix Čx one can equivalently compute the entropy of the eigenvalues of the kernel similarity matrix K. This approach results in the definition of the VENDI and RKE diversity scores: defines the family of VENDI scores as
VENDIA (X1, (XI,...,,,) x,) == exp Ha (k)
1
n
where . An denote the eigenvalues of the kernel matrix 1 K. Also, proposes the RKE
score, which is the special order-2 Renyi entropy, RKE(X,,.... x,) = exp(H2 K)). To compute
RKE without computing the eigenvalues, [2] points out the RKE score reduces to the Frobenius norm || || of the kernel matrix as follows:
RKE(X1,...... x,)
K
1 n
(xxxx,)²)
k
3.2 Shift-Invariant Kernels and Random Fourier Features
A kernel function k is called shift-invariant, if there exists a function K Rd + R such that x(x.x') = x(x x') for every x, E R.. Bochner's theorem proves that function Rd + IR will lead to shift-invariant kernel similarity score s(x- x') between x, x' if and only its Fourier transform & _Rd + R is non-negative everywhere (i.e, R(w) 2 0 for every w). Note that the Fourier transform R defined as
mxxx))x(((>>))) (2 / m((x) exp(-ii iw "x)xx
R(w)
Specifically, Bochner's theorem shows the Fourier transform a normalized shift- ihift-invariant kernel x(x,x') <(x-x'), where x<0) = 1, will be a probability density function (PDF). The framework of
random Fourier features (RFFs) [6 utilizes independent samples drawn from PDF to approximate
the kernel function. Here, given independent samples wi,-.., w, R, form the following proxy
feature map dr RR 2
(x) = Vr [os(w]] x), sin x)..... .., cos(ww, sin(w,,
As demonstrated in 16 42 the 2r-dimensional proxy map di can approximate the kernel function as (xxx x') E cos (w x) cos(ww T x') + sin(w" x) sin [x)) x = (x)
(x)))x (x))
4 Computational Complexity of VENDI & RKE Scores
As discussed, computing RKE and general VENDI, scores requires computing the order-a entropy of kernel matrix -.. Using the standard definition of a-norm ||vl|o (Ein |o 1/a we
observe that the computation of VENDI, score is equivalent to computing the n-norm | of the n-dimensional eigenvalue vector = [Ar..... An] where \ 2 ... 2 An are the sorted cigenvalues of the normalized kernel matrix 1 K.
In the following theorem, we prove that except order a = 2, which is the RKE score, computing any other VENDI,, score is at least as expensive computing the product of two matrices. Therefore, the theorem suggests that the computational complexity of every member the VENDI family is lower-bounded by 2(3223372) which is the least known cost of multiplying n n matrices.
In the theorem, we suppose B is any fixed set of "basis" functions. A circuit C is directed acyclic graph each of whose internal nodes is labeled by gate coming from set B. A subset of gates are
5

---

designated as outputs of C. A circuit with n source nodes and m outputs computes a function from In ! by evaluating the gate at each internal gate topologiial order. The size of circuit is the number of gates. Also, VB is the basis consisting of the gradients of all functions in B. We will provide the proof of the theorems the Appendix.
( for a * 2 is computable by a circuit C of size s(n) over basis B. then n n matrices can be multiplied by a circuit C of size (()(n)) over basis B U (+,
Theorem 1. If & (K)
Remark 1. The smallest known circuits for multiplying X n matrices have size ( where = 2.372. Despite tremendous research efforts only minor improvements have been obtained in recent years. There is evidence that w' is bounded away from 2 for certain classes of circuits [43 44).
In contrast, S is computable in quadratic time (n²)) in the basis B {x,+.log)
The above discussion indicates that except the RKE(X)....., .x)) i.e. order-2 Renyi entropy, whose computational complexity is quadratically growing with sample size (n²²). the other members of the VENDI family VENDI, would have a super- quadratta complexity on the order of ( In practice, the computation of VENDI scores is performed by the eigendecomposition of the kernel matrix that requires (nn)) computations for precise computation and O M) computations using a randomized projection onto M-dimensional space [I
5 A Scalable Fourier-based Method for Computing Kernel Entropy Scores
As we showed earlier, the complexity of computing RKE and VENDI scores are at least quadratically growing with the sample size n. The super-linear growth of the scores' complexity with sample size n can hinder their application to large-scale datasets and generative models with potentially hundreds of sample types. In such cases, a proper entropy estimation should be performed over potentially hundreds of thousands of data, where the quadratic complexity of the scores would be a significant barrier toward their accurate estimation.
Here, we consider a shift-invariann kernel matrix k(x, x') <(x - x') where k(0) and propose applying the random Fourier features (RFF) framework [16] to perform an efficient approximation of the RKE and VENDI scores. To do this, we utilize the Fourier transform R that, according to
Bochner's 's theorem, is a valid PDF, and we independently generate Wi....; wr R. Note that in the case of the Gaussian kernel (Ga)siiano2)), the corresponding PDF will be an isotropic Gaussian (O, J Ia) with zero mean and covariance matrix 2 Lg. Then, we consider the RFF proxy feature map o R2d_ - RR22 as defined in ① and define the proxy kernel covariance matrix Cx E R22x2r
Čx,-
o (xi) oe (x:))
Note that the 2r x 2r matrix Čx,r has the same non-zero eigenvalues the n RFF proxy kemell matrix 1 K,, and therefore can be utilized to approximate the eigenvalues of the original n x n kernel matrix I K. Therefore, we propose the Fourier-basee Kernel Entropy Approximation (FKEA) method to approximate the RKE and VENDI, scores as follows:
FKEA-RKE (XI...,, ......,,,)............... = exp( = -1(xxrI1F
(3)
2r
FKEA-VENDI, (x (x....,,,,)) exp((H.
(4)
Note that in the above, X denotes the ith eigenvalue of the 2r x 2r matrix Čx,-. We remark that the computation of both FKEA-RKE and FKEA-VENDI, can be done by a stochastic algorithm which computes the proxy covariance matrix 2 by summing the sample-based 2r x 2r matrix terms, and then computing the resulting matrix's Frobenius norm for RKE score or all the 2r matrix's 's presents the steps of the FKEA method where the computation needed for the proxy kernel covariance matrix is ((n) and grows only linearly with sample size n.
eigenvalues for general VENDI, with & 2. Algorithm I
Therefore, to show the FKEA method's scalability, we need to bound the required RFF size 2r for an accurate approximation of the original n x kernel matrix. The following theorem proves that the needed feature size will be O( log R for an e-accurate approximations of the matrix's 's eigenspace.

---

Algorithm FKEA Algorithm for Computing VENDI and RKE reference-free scores
1: Input: datapoints (01,---.5 --l1------0 3.n kernel bandwidth a², RFF dimension
2: Draw i.i.d. samples wi...., W, ~ R.
For Gaussian Kernel ((O, dz Ta)
3: Initialize the covariance matrix C
4: Compute the covariance matrix:
5: for 1 to n2 do
6: Compute the RFF feature for zy;
1 Vr
(z,)
cos(ww] x4), sin(w] "., cos(w,,a.)). sin(w, 2x1]
7: Update C:
Ō 3 (z))))))))))))))))))))))))))))))
n
8: end for
9: Perform eigendecomposition on the covariance matrix:
A2r ;
Eigendecomposition(()
10: Compute VENDI and RKE metrics using the eigenvalues λz,... Aar
Theorem 2. Consider a shift-invariant kernel k(x, x') = x(x - x') where k(0) = Suppose
Wy W of the normalized kernel matrix 1 1K [k(, x x))] Also, consider the eigenvalues of X 2
R are independently drawn from PDF R. Let λ1 2 ....Z An be the sorted eigenvalues
n.
Xer of random matrix Cx,r with their corresponding eigenvectors Vi,...; V2r- Let X3 for every > 2r. Then, for every O, the following holds with probability at least 8:
Ar 21)² 18oo((n/22
and
Σ
1 Kv. X. $
32 logloo///// (2/)
where Vi = Ej=1 sin V22 X V2j + cos V2 22 ( is the ith proxy eigenvector for 1 K.
n
Corollary 1. In the setting of Theorem 2 the following approximation guarantees hold for RKE and VENDI, scores
For every VENDI, with a 2 2, including RKE for a = 2, the following dimension-inneppeddent bound holds with probability least 1 8:
8 log(nn/25)
FKEA-VENDI, (xI....,x.)
VENDI, (x1, (x..
For every VENDI, with 1 1 a < 2, assuming a finite dimension for the kernel feature map dim )) m, the following bound holds with probability at least 1 8:
Slog(n (
PKEA-VENDL, (XI....,
VENDI, ( (XI....,
m
Remark 2. According to the theoretical results in (45), the top-t eigenvectors of kernel covariance matrix Cx will correspond the mean of the modes of a mixture distribution with t well-separable modes. Theorem shows for every S i < 2r, FKEA provides the proxy score function ü; Rdd - IR
that assigns likelihood score for an input x to belong to the ith identified mode:
(x()) = Σ sin V
(V2jx) V2j. + cos (
V2j-1.1
(5)
Therefore, one can compute the above FKEA -bused score for each of the 2r eigenvectors over sample set, and use the samples with the highest scores according to every ūn characterize the sample cluster captured by the FKEA method.

---

0
2
VINDA 5.8
VIND
RCS
250
50-
6
7
200
PROAVEVOLLL
10-
-0-
a
150
25
6
7
8
100
10-
2
0-
Colored Dataset
number tabels
Top 10 FKEA Identified Modes in Colored MNIST Dataset
4
4
4
4
4
9
9
9
9
8
8
8
8
8
4
4
4
4
4
9
9
9
8
8
8
8
8
4
4
4
9
9
9
9
8
8
8
8
4
4
9
9
9
9
8
8
8
8
8
Mode #1
Mode #2
Mode #3
Mode #4
Mode #5
6
6
6
6
6
2
2
2
2
2
7 7
7
7
3
3
6
6
6
6
2
2
2
7
7
7
7
7
3
3
3
6
6
6
6
2
2 2
2
7
7
7
7
7 7
3
3
6
6
6
6
J
2
2
2
7
7
7
7
Mode
Mode
Mode
Mode
Figure 2: RFF-based identified clusters used in FKEA Evaluation in single-colored MNIST [6 dataset with pixel embedding, Fourier feature dimension 2r 4000 and bandwidth graphs indicate increase in FKEA RKE/VENDI diversity metrics with increasing number of labels.
The
Table Time complexity for FKEA and mon-FKEA based metries (RKE and VENDI) on ImageNet dataset with V2 embedding. Computation of VENDI and RKE on 40k+ samples are omitted due memory overflow during metric computation.
Time
Metric
n-10k
n-20k
B-40k
n-50k
n- n-100k
n-250k
n-10k
n-20k
n-30k
n--4kk
n-50k
n- n-100k
n-250k
PREA-KKE
7
16
25
34
43
87
238
37
78
120
162
203
433
1138
PREA VENDI
19
45
104
267
89
130
173
213
459
1236
RKE
217
1324
4007
218
1330
VENDI
286
1774
5488
287
1780
5502
Numerical Results
We evaluated the FKEA method on several image, text, and video datasets to assess its performance in quantifying diversity in different domains. In the experiments, we computed the empirical covariance matrix of 2r-dimensional Fourier features with Gaussian kernel with bandwidth parameter tuned for each dataset, and then applied FKEA approximation for the VENDI, VENDII,s and the RKE (same VENDI,) scores. An algorithm to compute these scores is presented in Algorithm T Experiments were conducted on RTX3090 GPUs. We interpreted the modes identified by FKEA entropy- based diversity evaluation using the eigenvectors of the proxy covariance matrix discussed in Remark 2 For each eigenvector, we presented the training data with maximum eigenfunction values corresponding to the eigenvector.
Time Complexity of FKEA metrics. To highlight the computational advantages of transitioning to FKEA, Table presents comparison of the metric computations for VENDI and RKE on the ImageNet dataset, with sample sizes ranging from 10k to 250k. Our results show that VENDI and RKE become computationally intractable due to memory overflow. In contrast, the FKEA method efficiently scales up to n = 250k samples, maintaining optimal computational time.
Experimental Results on Image Data. To investigate the FKEA method's 's diversity evaluation in settings where we know the ground-truth clusters and their quantity, we simulated an experiment on the colored MNIST 46 data with the images of 10 colored digits as shown in Figure 2 We evaluated the FKEA approximations of the diversity scores while including samples from digits for t e ( 10}. The plots in Figure 2 indicate the increasing trend of the scores and FKEA's tight

---

4000
FREA PREA- FXEA-KKE (VENDI-2
1000
FREA PEA-NEAVEEEO
FKEA PEEEAEENIILLLL
5000
2000
500
2500
PREAVENDI-1.
PKEAVENDI-155
-0 FKEA-RKE (VENDI-2 2]
200
600
8001000
200 400 600 8001000
200 400 600 8001000
Top 4 FKEA Identified Modes
Mode #1
Mode #2
Mode #3
Mode #4
Figure 3: CFF-based identified clusters used in FKEA Evaluation in ImageNet dataset with Dino embedding, Fourier feature dimension 2r
16k and Gaussian Kernel bandwidth o = 25. The graphs indicate increase in FKEA diversity metrics with increasing number of labels per 50k samples.
PKEA-RKE (VENDI-2)
FREA-VEND-11
2000
PKEA-VENDI-1 0
FKEA-EEND-1.5
300
PKEAVENDI-15
75
FKEA-RKE (VENDE-2)
200
50
1000
100
25
0.5
0.5
truncation factor
truncation factor
truncation factor
0.4
V= 0.6
V =
1 ==0
Figure 4: FKEA metries behavior under different truncation factor vb StyleGAN3 47 FFHQ samples.
generated
approximations of the scores. Also, we show the top-20 training samples with the highest scores according to the t0p-10 FKEA eigenvectors, showing the method captured the ground-trui -ruth clusters.
We conducted experiment the ImageNet data monitor the scores' behavior evaluated for 50k samples from an increasing number of ImageNet labels. Figure shows the increasing trend of the scores well as the top-9 samples representing the top-4 identified clusters used for the entropy calculation. Also, Figure presents the FKEA approximated entropy scores with different truncation factors in StyleGAN3 [47] on 30k generated data for each truncation factor, showing the increasing diversity scores with the truncation factor. We defer discussing the results on AFHQ 48 MS-COCO 49 F-MNIST 50 datasets the Appendix.
Table 2: Top synthetic countries dataset modes with rext-embedding----ae embedding, Fourier features dim 2r = 8000 and Gaussian Kernel bandwidth o = 0.9. The table summarises the mentions of each country in the top 100 paragraphs identified for the eigenvectors corresponding to each mode.
Mode #1
Mode #2
Mode #3
Mode #4
Mode #5
Mode #6
Burkina Faso
34%
Argentina 77%
Azerbuijan
100%
Cambodia
94%
Belarus
100%
Bolivia 97%
Benin
23%
Chile
23%
Afghamistan
6%
Iscaado
3%
22%
Burundi
13%
Cameroon
8%

---

1000
FKEA FREAARR RICE (VENDI-2)
PREA PREA PREA-ENDDD
750
FREA: RREA- PREA-EEND-11
20
50
500
10-
250
25
PREAVENDI-10
PREAVENDI-15
FREA-RKE (VENDI-2)
20
40
60
80
100
0
20
40
60
80
100
20
40
60
80
100
number countries
number of countries
number countries
Figure 5: FKEA diversity metrics with the increasing number of countries in the synthetic dataset.
1000
FKEE RKEEVENDI-2)
FREA VENDI-N
PKEA-VEND-1.5
500
100
25
PKEAVENDI-100
FKEA-VEND--1.5
-0
FKEA-RKE (VENDI-2)
0
0
0-L
25
50
75
100
25
50
75
100
25
50
75
100
(a) Mode #1
(b) Mode #2
(c) Mode #3
(d) Mode #4
Figure 6: RFF-based identified clusters used in FKEA evaluation in UCF101 dataset with 13D embedding. The graphs indicate an increase in FKEA diversity metrics with more classes.
Experimental Results Text and Video Data. To perform experiments the text data with known clustering ground-truth, we generated 500,000 paragraphs using GPT-3.5 [51] about 100 randomly selected countries (5k samples per country). In the experiments, the text embedding used was -ett--mbedding-----e 3-large [52 53 51]. We evaluated the diversity scores over data subsets of size 50k with different numbers of mentioned countries. Figure
the growing trend of the diversity scores when including more countries. The figure also shows the countries mentioned in the top-6 modes provided by FKEA-based principal eigenvectors, which shows the RFF-based clustering of countries correlates with their continent and geographical location. We discuss the numerical results on non-synthetic text datasets, Wikipedia, CNN//ailymail [54 [55] CMU Movie Corpus [56], the Appendix.
For video data experiments, we considered two standard video datasets, UCF101 57 and Kinetics- 400 [58]. Following the video evaluation literature (59 60). we used the I3D pre-trained model [61] as embedding, which maps video sample increasing the number of video classes of test samples led to an increase in the FKEA approximated diversity metrics. Also, while the samples identified for the first identified cluster look broad, the next modes seemed more specific. We discuss the results of the Kinetics dataset in the Appendix.
0024-dimensional vector. As shown Figure 6
7 Conclusion
In this work, we proposed the Fourier-based FKEA method to efficiently approximate the kernel-basse entropy scores VENDI, and RKE scores. The application of FKEA results in scalable reference- free evaluation generative models, which could be utilized in applications where no reference data is available for evaluation. A future direction to work is study the sample complexity of the mutrix-based entropy scores and the FKEA's approximation under high-dimensional kernel feature maps, e.g. the Gaussian kerne.. Also, analyzing the role of feature embedding in the method's application to text and video data would be interesting for future exploration.
10

---

Acknowledgments
The work of Farzan Farnia is partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China, Project 14209920, and is partially supported by CUHK Direct Research Grant with CUHK Project No. 4055164. Xuenan Cao's work is supported by grant from the Research Grants Council of the Hong Kong Special Administrative Region, China, Project 14602223. Andrej Bogdanov's work is supported by an NSERC Discovery Grant.
References
[1] Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning, In Transactions on Machine Learning Research, 2023.
[2] Mohammad Jalali, Cheuk Ting Li, and Farzan Farnia. An information-theooriicc evaluation of generative models in learning multi-modal distributions. In Advances in Neural Information Processing Systems, volume 36, pages 9931-9943, 2023.
[3] Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2013.
[4] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, (116840-6551) 2020.
[6] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. In International Journal of Computer Vision (IJC),, number 3. pages 21 1-22, 2015.
[7] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2018
[8] Mikolaj Binkkoskii, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In International Conference on Learning Representations, 2018.
[9] Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly, Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[10] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[11] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh. Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of ICML '2, pages 7176-7185 JMLR.org, 2020.
[12] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training GANs. In Advances Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[13] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXLL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
[14] Amey Pasarkar and Adji Bousso Dieng. Cousins of the vendi score: A family of similarity- diversity metrics for science and machine learning. International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

---

[15] Yuxin Dong, Tieliang Gong, Shujian Yu, and Chen Li. Optimal randomized approximations for matrix-based rényi's entropy. IEEE Transactions on Information Theory, 2023.
[16] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
[17] Ahmed M. Alaa, Boris van Breugel, Evgeny Saveliev, and Mihaela van der Schaar. How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models, July 2022. arXiv:2102.08921 [es, stat].
[18) Marco Jiralerspong, Joey Bose, Ian Gemp, Chongli Qin, Yoram Bachrach, and Gauthier Gidel. Feature likelihood score: Evaluating the generalization of generative models using samples. Advances in Neural Information Processing Systems, 36, 2024.
[19] Jiyeon Han, Hwanil Choi, Yunjey Choi, Junho Kim, Jung-Woo Ha, and Jaesik Choi. Rarity score: A new metric to evaluate the uncommonness of synthesized images. arXiv preprint 2rXiv:2206.08549, 2022.
[20] George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. In Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 3732-3784. Curran Associates, Inc., 2023.
[21] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnacve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. In Transactions on Machine Learning Research, 2023.
[22] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The Role of ImageNet Classes in Fréchet Inception Distance. September 2022.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, pages 8748-8763. arXiv, February 2021. arXiv:2103.00020 [cs].
[24] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In Advances in Neural Information Processing Systems, volume 33, pages 9912-9924, Curran Associates, Inc., 2020.
[25] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818 8-266. ISSN: 1063-6919.
[26] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pages 311-318. Association for Computational Linguistics, 2002.
[27] Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 18, pages 1097-1100. Association for Computing Machinery, 2018
[28] Vishakh Padmakumar and He He. Does Writing with Language Models Reduce Content Diversity?, March 2024. arXiv:2309.05199 [cs].
12

---

[29] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zey, and Mike Lewis. Contrastive Decoding: Open-endee Text Generation as Optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceediings of the 61st Annual Meeting of the Association for Computational Linguisties Volume Long Papers), pages 12286-12312, Toronto, Canada, July 2023. Association for Computational Linguistics.
[30] Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Locally Typical Sampling. Transactions of the Association for Computational Linguistics, 11:102-121, 2023. Place: Cambridge, MA Publisher: MIT Press.
[31] Chin Yew Lin and Franz Josef Och. Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguisties (ACL-004) pages 605-612, Barcelona, Spain, July 2004.
[32] Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. Jointly Measuring Diversity and Quality in Text Generation Models. arXiv, May 2019. arXiv:1904.0399 [cs, stat].
[33] Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa Siu, Byron C. Wallace, and Ani Nenkova. Standardizing the measurement of text diversity: A tool and a comparative analysis of scores.
[34] Bernhard Schölkopf, Alexander Smola, and Klaus- Müller. Nonlinear Component Analysis as a Kernel Eigenvalue Problem. Neural Computation, 1095):129--131, July 1998. Conference Name: Neural Computation.
[35] Yoshua Bengio, Pascal Vincent, Jean-Frangean Paiement, O Delalleau, M Ouimet, and N LeRoux. Learning eigenfunctions of similarity: linking spectral clustering and kernel pca. Technical report, Technical Report 1232, Departement d' d'Informatique Recherche Oprationnelle 2003.
[36] Yoshua Bengio, Pascal Vincent, Jean-Franqee Paiement, Olivier Delalleau, Marie Ouimet, and Nicolas Le Roux. Spectral clustering and kernel PCA learning eigenfunctions, volume 1239. Citeseer, 2003.
[37] Radha Chitta, Rong Jin, and Anil K Jain. Efficient kernel clustering using random fourier features. In 2012 IEEE 12th International Conference on Data Mining, pages 161-170. IEEE, 2012.
[38] Mina Ghashami, Daniel J Perry, and Jeff Phillips. Streaming kernel principal component analysis. In Artificial intelligence and statisties, pages 1365-1374. PMLR, 2016.
[39] Enayat Ullah, Poorya Mianjy, Teodor Vanislavov Marinov, and Raman Arora. Streaming kernel pea with o( (V/n) random features. Advances in Neural Information Processing Systems, 31, 2018.
[40] Bharath Sriperumbudur and Nicholas Sterge. Approximate kernel pea: Computational versus statistical trade-off. The Annals of Statistics, 50(5):2713-2747 2022.
[41] Daniel Gedon, Antônio H Ribeiro, Niklas Wahlström, and Thomas B Schön. Invertible kemne pea with random fourier features. IEEE Signal Processing Letters, 30:563-567, 2023.
[42] Danica J. Sutherland and Jeff Schneider. On the Error of Random Fourier Features. In Uncertainty in Artificial Intelligence (UAI) 2015. arxiv, June 2015. 106.021506 [cs, stat].
[43] Josh Alman. Limits on the universal method for matrix multiplication. Theory of Computing, 17(1):1-30, 2021.
[44] Matthias Christandl, Péter Vrana, and Jeroen Zuiddam. Barriers for fast matrix multiplication from inveveriiiility. Theory of Computing, 17(2):1-32. 2021.
[45] Jingwei Zhang, Cheuk Ting Li, and Farzan Farnia. An interpretable evaluation entropy-based novelty of generative models. In International Conference Machine Learning (ICML (024).
13

---

[46] Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. In IEEE Signal Processing Magazine, volume 29, pages 141-142, 2012. Conference Name: IEEE Signal Processing Magazine.
[47] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Advances in Neural Information Processing Systems, volume 34, pages 852-863. Curran Associates, Inc.
[48] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wir Woo Ha. Stargan v2: Diverse image synthesis for multiple domains, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[49] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, European Conference Computer Vision (ECCV), volume 8693, pages 740-755. Springer International Publishing, 2014. Book Title: Computer Vision ECCV 2014 Series Title: Lecture Notes in Computer Science.
[50] Han Xiao, Kashif Rasul, and Roland Vollgraf. fishion-mniste a novel image dataset for benchmarking machine learning algorithms, 2017.
[51] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33. pages 1877-1901. Curran Associates, Inc.
[52] OpenAL. text-embedding--Lee embeddings 2024.
http:://platformoooeaaa.commm cot//ccs/mooommel......
[53] OpenAl. GPT-4 Technical Report, March 2024. arXiv:2303.08774 [cs].
[54] Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, pages 1693-1701, 2015.
[55] Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[56] Brendan O'Connor David Bamman and Noah A. Smith. Learning latent personas of film characters. In ACL 2013, 2013.
[57] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild, 2012.
[58] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017.
[59] Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128( (28(11)):566266 Nov 2020.
[60] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric challenges, 2019.
14

---

[61] João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724 4733, 2017.
[62] S. Linnainmaa. The representation of the cumulative rounding error of an algorithm taylor expansion of the local rounding errors. Master's thesis, University of Helsinki, 1970.
[63] W. Baur and Strassen. The complexity partial derivatives. Theoretical Computer Science, 22:317 330, 1983.
[64] Alan J Hoffman and Helmut W Wielandt. The variation of the spectrum of normal matrix. In Selected Papers Of Alan J Hoffmann With Commentary, pages 1 118-120. World Scientific, 2003,
[65] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Advances Neural Information Processing Systems, volume 34, pages 8780-8794. Curran Associates, Inc.
[66] Andrew Brock, Jeff Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis.
[67] William Peebles and Saining Xie. Scalable diffusion models with transformers.
[68] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[69] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 06674-10685. IEEE Computer Society.
[70] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[71] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11523-11533 2022.
[72] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings, volume his/2201.002277 2022.
[73] Rewon Child. Very deep VAEs generalize autoregressive models and can outperform them images.
[74] Ceyuan Yang. Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. In Advances in Neural Information Processing Systems, volume 34, pages
Curran Associates, Inc
[75] Ricardo Campos, Vitor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Jatowt. YAKE! keyword extraction from single documents using multiple local features. In Information Sciences, volume 509, pages 257-289, 2020.
15

---

A Proofs
A.1 Proof of Theorem
The proof of Theorem 1 combines third ingredients. The first is the relation between the circuit size of function C' and of partial derivatives VC = /oDC/an....,800 ,0C/0r,
Lemma 1. The function VC has a circuit over basis VBU {+, whose size within a constant factor of the size of C.
Lemma D is a feature of the backpropagation algorithm 62 This is linear-time algorithm for constructing circuit for VC given the circuit C' as input. In contrast, the forward propagation algorithm allows efficient computation of single (partial) derivative even for circuits with multivalued outputs, giving the second ingredient:
Lemma 2. Let C' be circuit over basis B and t be an input to C. There exists circuit that computes the derivative 0g/ jt for every gate g of over basis B {+, X} whose size within a constant factor of the size of C.
The last ingredient is the following identity. For a scalar function f over the complex numbers and matrix X diagonalizable UAU define f(X) to be the function U fA)U" where is applied entry-wise to the diagonal matrix
Lemma For every S that analytic over an open domain containing all sufficiently large complex numbers and every matrix X whose spectrum is contained in S VTr(((X)) fX).
We first illustrate the proof in the special case when ||] is within the radius of convergence of f- Namely, fa) is represented by the absolutely convergent series (()))()))* (D)/*///a /k! for all | S p.
Then f(X) = Σ ((*) (0) /k! assuming || S p. By linearity (and using the fact that derivatives
preserve radius of convergence) it is sufficient to show that
VTrX* dx*
(6)
dx
which can be verified by explicit calculation: Both sides equal kx*=1, This is sufficient to establish Theorem for all integer a > 2.
Proof of Lemma The Cauchy integral formula for matrices yields the representation
s(X) 2 [ [==)(=(-XX(])())] xz=
for any closed curve whose interior contains the spectrum of X. As (zl X)- is continuous along C, we can write
2 / [x=)V[((llxx----
1
vIrf(X)
Choosing C' to be a circle of radius p larger than the spectral norm of X, for all a of magnitude p we have the identity
(x1 (X1-X)-1- -1- -1 -1-
k x
k=
As the series Σ ~VTx k x converges absolutely in spectral norm, using (6) we kz
obtain the identity VTr(zl - X) x)= d(zl - X)- /dX, namely the lemma holds for the function
fX) (zI (x-x -)- Plugging into 7 and exchanging the order of integration and derivation proves the lemma.
Proof of Theorem 7 Assume Trpx (resp., -Trplog p) has circuit size s(d). By Lemma and Lemma 3 VTTpp" apa- (resp., - -TTrp log p log p - 1/ /n 2) has circuit size ((s(d). For every symmetric matrix X and sufficiently small f, the matrix = I + tX is positive semi-definite. By Lemma 2 the _ -valued function 8 p/at² has circuit size (s The value of this function at
16

---

== is a(a - 1) 2)X²² (resp.. X²), namely the square of the input matrix X up to constant. Finally, computing the product AB reduces squaring the symmetric matrix
A
T
A B
A.2 Proof of Theorem 2
Assuming that the shift-invariant kernel k(x, x(x x') is normalized (i.e. x(O) 1), then the Fourier transform R is a valid PDF according to Bochner's theorem and also an even function because takes real values. Then, we have
k xxx,x)
(x xx)
Ke (w) exp w T (x - x') x'
(b) Ks (w) cos cos(w (w (x - dw
Ew (ws(w)) (x (x-x))
BuwR
(wss(w)
x) cos ( Tuxx))
x') + sin sin(wTx x) sin [x)]
Here, (a) comes from the synthesis property of the Fourier transform. (b) holds since Ri is an function, resulting in a zero imaginary term in the Fourier synthesis.
Therefore, since |cos(w ( [T)) < 1 for all and y, one can apply Hoeffding's inequality to show for independently drawn Wi...., Wp lid R the following probably correct bound holds:
( Σ (ws (x x)) Ewna cos( (w T (x x))) 20) s 2exp(-") rr?
2
Therefore, as the identity cos(a = cas(a) cos(b) + sin(a) sin(b) reveals + Ei-1 cos(w) (x x')) = (x) (x'), the above bound can be rewritten as
(((1 (xxx(x) (xx (x') x(x,x') 26) S 22exp'
rE² 2
P
Also, (x, = (x) (x') is by definition normalized kernel, implying that
Vx E Rd (x) xx) k(x, x) 0.
As result, one can apply the union bound to combine the above inequalities and show for every sample set X1..... Xn:
re
P max ISijiS
Tx,
)2) 2 e2) S2
exp
Considering the normalized kernel matrix 1 K
[K(x;, xj)] isujsn and the proxy normalized
kernel matrix 1 k = 2 (x,) Hr (x,)] Si the above inequality implies that
(1 [1|=_ k - - Kl| 2 n2
2
11
re
exp
71
n
2
2
p - 2 $) <
n²
re
2
(8) we can translate the above bound to the
exp
n
2
2
Leveraging the eigenvalue-perruubattin bound following for the sorted eigenvalues \ 2 ... 2 An of K and the sorted eigenvalues Àι 2 ... 2 X, of 1 K
64
Tr x,)2 < L K
17

---

which shows
p'
n²
rE²
PP
Σ
);)² 2 c)
exp
(9)
2
2
1=1
blog (log(1/20)
Defining 8
n² I exp
exp(-
rx²
i.e., E =
leads to
n
P
(A 11)2 5c) 2 1
(10)
Noting that the normalized proxy kernel matrix K and the proxy kernel covariance matrix Cx share identical non-zero eigenvalues together with the above bound finish the proof of Theorem 2 first part.
Concerning Theorem 2 s approximation guarantee for the eigenvectors, note that for each eigenvectors V; of the proxy kernel matrix n. K, the following holds:
1 KV, Av -
1 Kv, x. [ -..111
= || K - k ||,
+ +1 -1
Therefore, applying Young's inequality shows that
1 KV. 2v.1. - 2| - K- -
1
K)
v;
2(4 -11)²
71
n
n1
2
2Tr V (-K - 1 k) v.)
+ 2( 2.)2
n
= 2Tr V. (
1 K- - x)²)
+ 2(x - 21)2
which implies that
Kv: AoV. 1 2T Σ (. (- K- - +2 > (A+ x.)
n
(
1 x))2) + 2 > (7s - A1)2
2Tr
K
n
1=1
=
= k
1 _
+ > (1+ x1)
n
K
K
7
The above proves that
P
1
Kv.
1/ 26
<
P
1
1
_
1
K
2
2
n
2
71
n
F
n²
TE2
exp
8
Therefore, considering the provided definition 8 3 exp will have the following which completes the proof:
n
7/7
r62
i.e., 2e
32 32 (n/26)
we
Σ i=1
n
L kv
2vil Mivv.-A,ii//
22)
2 6.
18

---

Number of ImageNet Samples = 10k
Mode #1
Mode #2
Mode #3
Mode #4
Number of ImageNet Samples = 50k
Mode #1
Mode
Mode #3
Mode #4
Number of ImageNet Samples = 100k
Mode #1
Mode #2
Mode #3
Mode #4
Number of ImageNet Samples = 250k
Mode #1
Mode #2
Mode #3
Mode #4
Figure 7: RFF-based identified clusters used in FKEA Evaluation in ImageNet dataset with Dino embeddings and bandwidth 25 at varying number of samples n
19

---

Top 8 FKEA Identified Modes FFHQ Dataset
Mode #1
Mode #2
Mode #3
Mode #4
Mode #5
Mode #6
Mode #7
Mode #8
Figure 8: RFF-b identified clusters used in FKEA Evaluation in FFHQ dataset with V2 embeddings and bandwidth 20
B Limitations
Incompatibility with non shift-invariant kernels. Our analysis targets shift-invariant kernel, which does not apply to a general kernel function, such as polynomial kernels. In practice, many ML algorithms rely simpler kernels that may not have the shift-invarian property. Due to specifics of FKEA framework, we cannot directly extend the work to such kernels. We leave the framework's extension to other kernel functions for future studies.
Reliance on Embeddings. FKEA clustering and diversity assessment metrics rely on the quality of the underlying embedding space. Depending on the training and pre- traitrann datasets, the semantic clustering properties may change. We leave in-depth study of embedding space behavior for future research.
C Additional Numerical Results
C.1 Real Image Dataset Modes
This section details the results of cluster analyses conducted on various real-world datasets, including FFHQ, AFHQ, MSCOCO, and Fashion-MNIST. Each dataset's 's results are organized into clusters identified by RFF method in FKEA evaluation.
C.2 The effect of number of datapoints on clustering results with FKEA
In this section, we evaluate the quality clusters obtained from the ImageNet dataset as the number of samples varies. Specifically, we compare clustering results for 10k, 50k, 100k, and 250k samples. Figure fillutrallss the first four modes derived from the FKEA framework.
At n = 10k, the clusters exhibit noise and often merge unrelated modes, as seen in Mode 2, where elephants and foxes appear within the same cluster. As increases, the clustering quality improves, becoming more coherent and meaningful. This trend is particularly evident in Modes and 2, where the clusters more accurately reflect distinct semantic groups
These findings highlight the importance of scaling VENDI and RKE scores, as computational overhead becomes critical factor in assessing the diversity of generative models. Scaling these
20

---

Top 8 FKEA Identified Modes in AFHQ Dataset
Mode
#3
Mode #4
Mode
Mode #7
Mode
Figure 9: RFF-based identified clusters used in FKEA Evaluation in AFHQ dataset with Dino V2
embeddings and bandwidth =
Top 8 FKEA Identified Modes in MSCOCO Dataset
Mode
Mode #3
Mode #4
Mode
Mode #7
Mode
Figure 10: RFF-based identified clusters used in FKEA Evaluation in Microsoft COCO dataset with Dino embeddings and bandwidth o 22
21

---

Top 8 FKEA Identified Modes in F-MNIST Dataset
0
0
M
0
Mode
Mode #2
Mode #3
Mode #4
1
Mode
Mode
Mode #7
Mode
Figure 11: RFF-based identified clusters used in FKEA Evaluation in CESHION-MNISTE 50 dataset with pixel embeddings and bandwidth a 15
Top 8 FKEA Identified Modes in Color F-MNIST Dataset
Mode
Mode
Mode #3
Mode #4
Mode
Mode #7
Mode
Figure 12:
RFF-based identified clusters used in FKEA Evaluation in colored FASHION-MNIST 50 dataset with pixel embeddings and bandwidth a 4.5
22

---

Table 3: Evaluated scores for ImageNet generative models. The Gaussian kernel bandwidth parameter
chosen for RKE, VENDI, FKEA- VENDI and PKEA-RKE is = 25 and Fourier features dimension
2r 164. The scores were obtained by running the GutHub of 20) on pre-generated 50k samples.
Method
f
Precision
Density
Coverage
FKEA VENDI-1
Dutaset (100k)
9176.9
996.7
ADM 65
542.6
11.12
0.78
0.79
0.88
0.89
8360.3
633.4
ADMG
659.3
5.63
0.87
0.84
0.85
8524.2
811.5
ADMG-ADMU
701.6
4.78
0.90
0.73
0.96
8577.6
839.8
BigGAN
66
696.4
7.91
0.81
0.44
0.99
0.57
7120.5
492.4
DiT-XL-2 89
743.2
3.56
0.92
0.84
1.16
0.97
8626.5
855.8
GigaGAN
68
678.8
4.29
0.89
0.74
0.74
0.70
8432.5
671.6
1.DM 69
734.4
4.75
0.93
0.76
1.04
0.93
8573.7
811.9
Mask-GIT
717.4
5.66
0.91
8557.4
759.5
RQ- Transformer
71
558.3
9.57
0.80
0.76
0.77
0.59
8078.4
512.1
StyleGAN-XI.
72
675.4
4.34
0.89
8171.9
703.5
metrics allows for a more efficient evaluation, especially when dealing with large datasets and high sample counts.
C.3
Comparison between Generative Models ImageNet dataset
In this section we report the FKEA scores for various generative models on ImageNet dataset. Table evvallates the diversity scores of various ImageNet GAN models using the FKEA method applied to
3
VENDI-1 and RKE, with potential extension to the entire VENDI family. The comparison includes baseline diversity metries such Inception Score 112 FID Improved Precision/Recall [10]. and
Density/Coverage
C.4 Synthetic Image Dataset Modes
In addition running clustering on ImageNet dataset, also applied FKEA with varying Gaussian Kernel bandwidth parameter a to other datasets. The results are presented for FFHQ (Figure R).
AFHQ (Figure 9) Microsoft COCO (Figure 10) and Mono/Color versions of F-MNIST (00 (Figures
and 22] to top 8 modes.
The experimental setup is similar to figure 3 with the only change is optimised bandwidth for each dataset, since datasets differ number typicality of the samples.
C,5 Effect of other embeddings on FKEA clustering
Even though D2 is primary embedding in our experimental settings, we acknowldge the use of other embedding models such SwAV (4 and CLIP The resulting clusters differ from original DinoV2 clusters and require separate bandwidth parameter finetuning. In our experiments. SwAV embedding emphasizes object placement, such as animal grass or white backgrounds, as seen in Figure 17 CLIP on the other hand clusters by objects, such as birds/dogs/bugs, as seen in Figure [88 These results indicate that FKEA powered by other embeddings will slightly change the clustering features; however, it does not hinder the clustering performance of RFF based clustering with FKEA method.
C.6 Effeet of embeddings on score convergence
To highlight the compatibility of FKEA across diverse embedding spaces, we conducted convergence experiments on various text and image embeddings. Figure presents the convergence results the VENDI and RKE scores, comparing both FKEA and non-FKEA counterparts. Our findings show that the convergence remains consistent across different embedding spaces, demonstrating the robustness of the proposed method,
C.7 Text Dataset Modes
To understand the applicability and effectiveness of the FKEA method beyond images, we extended study to text datasets. We observed that clustering text data poses more challenging task
23

---

Top FKEA Identified Modes Generative Model LDM in FFI
Mode #1
Mode # 2
Mode # 3
Mode # 4
Mode # 5
Mode # 6
Mode # 7
Mode # 8
Figure 13: RFF-based identified clusters used in FKEA Evaluation of LDM 69 generative model in
FFHQ with DINOv2 embeddings and bandwidth a = 20
Top FKEA Identified Modes of Generative Model VDVAE in FFHQ
Mode #1
Mode #
Mode # 3
Mode H 4
Mode # 5
Mode # 6
Mode # 7
Mode # 8
Figure 14: RaF-based identified clusters used in FKEA Evaluation of VDVAE 73 generative model
in FFHQ with DINOv2 embeddings and bandwidth a = 20
24

---

Top FKEA Identified Modes Generative Model InsGen in FFI
Mode
Mode # 2
Mode # 3
Mode # 4
Mode #
Mode # 6
Mode # 7
Mode # 8
Figure 15: RFF-based identified clusters used in FKEA Evaluation of InsGen in FFHQ with DINOv2 embeddings and bandwidth a = 20
generative model
Top 8 FKEA Identified Modes of Generative Model StyleGAN-XL FFHQ
Mode
Mode #
Mode # 3
Mode Il 4
Mode #
Mode # 6
Mode # 7
Mode # 8
Figure 16: RFF-based identified clusters used in FKEA Evaluation of StyleGAN 72 generative model in FFHQ with DINOv2 embeddings and bandwidth o = 20
25

---

Top 8 FKEA Identified Modes in ImageNet Dataset with SwAV
Mode
#3
Mode #4
Mode
Mode #7
Mode
Figure 17: RFF-based identified clusters used in FKEA Evaluation of SwAV embedding ImageNet with bandwidth a 0.8
Top 8 FKEA Identified Modes in ImageNet Dataset with CLIP
Mode
Mode #3
Mode #4
Mode
Mode #7
Mode
Figure 18: RFF-based identified clusters used in FKEA Evaluation of CLIP embedding ImageNet with bandwidth a 7.0
26

---

text text-embeddinng--ae
text-embedding-3-ae 3-small
Gernini
BERT
250
200
50
200
20
40
60
80
100
20
40
60
80
100
0
20
40
60
80
100
20
40
60
00
100
10-
20
2.5
0-
0
0.0
100
20
60
100
40
80
20
60
100
countries
number countries
number countries
countries
(a) Diversity convergence on synthetic countries dataset across various text embeddings
DinoV2
CLIP
Inceptionv3
SWAV
2000
500
1000
VENDI-1.0 3.0
PKEA-END--10
600 800 1000
200
400 600
1000
200
400
800 1000
600
800 1000
50
10-
AKE AXE
200 400 600 800 1000
200 400 600 800 1000
200 400 600 800 1000
200 400 600 800 1000
number of classes
number of classes
number of classes
number of classes
(b) Diversity convergence on ImageNet dataset across various image embeddinns
Figure 19: Summary of diversity convergence with r 12000 and sample size n 20000.
Mode
Mode
Mode
Mode
Grosse Pointe
Ishkli
Gerson Garca
Ginugamesh (album)
2009 WPSL seuson
Mark Scharf
Khazora
Valentin Mogilny
Japonesque (album)
2012 Milwaukee...
Alexander McKee
Sis. Azerbaijun
Gerald Lehner (referee)
Documentaly
2020 San Antonio FC...
Clay Huffman
Zasun
Dmitri Nezhelev
EX-Gir
2020 HFX Wanderers FC...
Ravenna, Ohio
Zaravat
Grigori Ivanov
Indic 2000
2020 Sporting Kansas City...
C. M. Eddy Jr.
Bogut
Leonidas Morakis
Triangle (Perfumm album)
FC Tocson
Hornell, New York
Yakkakhona
Jos Luis Alonso Ber...
Waste Management (album)
2008 League
Larchmont, York
Yava, Tajikistan
Giovanni Gasperini
Fush Yu Mang
201 New Zealand Football...
Roben Hugue
Rkizyak
Mohamed Chah
Fantastipo (song)
2008009 Melhoume Victory FC...
General Hershy Bar
Khushikat
Louis Damaann
Xtoet
2012 Pittsbrrh Power season
Keywords
London
papulated places
players category
music video
American foothall
American History
Maplandia.com com Category
Association football
album
players Category
University
municipality
FIFA
studio
Foothall League
United States
village
World Cup
Records albums
League
World War
Osh Region
Summer Olympics
Singles Chart
League Soccer
Table 4: Top Wikipedia Dataset Modes with corresponding eigenvalues with text-embedding---lrry embeddings and bandwidth 1.0
compared to image data. This increased difficulty arises from the ambiguity in defining clear separability factors within text, contrast to the more visually distinguishable criteria in images. The process of evaluating text clusters not straightforward and often varies significantly based on human judgment and perception.
To visualise the results, we use YAKE 755 algorithm to extract the keywords in each text mode and present the identified unigram and bigram keywords. We demonstrate that the results hold for text datasets and identified clusters are meaningful.
Table 4 displays the identified clusters associated with Wikipedia article titles and keywords analyzed using the FKEA method. Identified mode I correlates most with historical figures/events/placce Mode 2 clusters smaller villages and rural regions together. Mode 3 is exclusively about people in sports, such athletes and referees. Mode 4 visualises various music bands and albums. Lastly, mode 5 presents the articles about sports events, such as football leagues.
Table outlines the largest modes identified within news dataset analyzed using the FKEA method, with a detailed focus on the content themes of each mode. The most dominant mode is associated with topics related to crime and police activities, indicating a frequent coverage area in the dataset.
27

---

Mode #1
Mode #2
Mode #3
Mode #4
Mode #5
police
President Obama
size
people
died
British police
Obama
year
severe weather
family
police officer
Barack Obama
weight
Death toll
plane crash
Police found
President
dress size
heavy rain
mother
family
White House
stone
Environment Agency
plane
found
Obama administration
Slimming World lost
million people
found
told police
Obama calls
rain
people
court
House
lose weight
flood warnings
children
arrested
United States
diet
people dead
hospital
model Table 5: Top 5 CNN//aillmai 3.0.0 [54][55] Dataset Modes with corresponding eigenvalues with sett-embedding---laaee embeddings and bandwidth 0.8.
home
Obama plan
people killed found dead
Mode
Mode #2
Mode
Mode
Mode
The House Tele..
Anand
Bring Your Smile Along
Chhota Bheem...
Walk Crooked Mile
Seems Like Old Times
Love You
The Girl Most Likely
Duck Amuck
Assignment Kill
Shadows and Fog
Toh Bant Pakki
Hips, Hips, Hooray!
Hare-Abian Nights
The Crime of the Century Murder Glen Athol Guns
Obsession
Abodh
Lady Be Good
Porky's Five and Ten
Milk Money
Khulay Aasman...
The Courtship of Eddic's...
Sock-a-Dood DeoodooDeDo
Very Bad Things
Kusthuri Maun
You Live and Learn
Buccaneer Bunny
Because of the Cats
Blame
Bell...
Chhaya
Dames
Hare Lift
The House Hate
The Miracle Man
Ych Dillagi
Painting the Clouds...
Scrap Happy Daffy
The Ace of Scotland Yard The World Gone Mad Firepower
The Sleeping Tiger
Deva
Pin Up Girl
Hic-cup Pup
The Scapegoat
Bhalohasa Bha...
Young to
The Gooly Gophers
Keywords
mystery
hindi film
musical theme songs
animation
crime
noir kidnap crime
romance
Tom & Jerry
murder
love
city romance
Spike
detective
marriage daughter
adventure
investigation
police
comedy
killer
Table 6: Top 5 CMU Movie Summary Corpus [56] Dataset Modes with corresponding eigenvalues
with text-embedding----oee genres to each movie in the first 100 paragraphs in each mode.
embeddings and bandwidth = 0.8. The table summarises the assigned
Mode is closely correlated with President Obama, reflecting a significant focus on political coverage. Mode pertains to dieting, which suggests a presence of health and lifestyle topics. Mode 4 is linked environmental disasters, highlighting the dataset's attention ecological and crisis-related news. Finally, Mode 5 deals with plane crash accidents, underscoring the coverage of major transportation incidents.
dablei]]eelineates the distribution of genres and production types within a dataset of movie summaries analyzed using the FKEA method. The first mode predominantly covers drama TV shows without focusing on any specific subtopic, indicating a broad categorization within this genre. From mode onwards, the features become more distinct and defined. Mode 2 specifically represents Bollywood movies, with a significant emphasis on the Romance genre. Mode 3 dedicated to clustering comedy shows. Mode 4 is exclusively associated with cartoons, evidenced by keywords such "Tom & Jerry". Lastly, mode 5 clusters together detective and crime fiction shows.
C.8 Video Dataset Modes
In this section, present additional experiments the Kinetics-400f53 video dataset. This dataset comprises 400 human action categories, each with a minimum of 400 video clips depicting the action. Similar to the video evaluation metrics, used the B3D pre-trained model which maps each video to 1024-vector feature. Figure [0] the first mode captured broader concepts while the other models focused on specific ones. Also, the plots indicate that increasing the number of classes from 40 to 400 results in an increase in the FKEA metrics.
28

---

FKEA-VENDI-1
2000
CKEA-VENDI-11. 1.5
1500
PKEA-RKEIVEND
1000
1000
1000
500
500
FKEA-VEND-1.5
FKEA-RKE (VENDI-2)
400
200
400
200
400
classes
(a) Mode
(b) Mode
Mode #3
(d) Mode #4
(e) Mode #5
(I) Mode #6
Figure 20: RFF-based identified clusters used in FKEA Evaluation in Kinetics-400 dataset with I3D embeddings. Plots indicate that increasing the number of classes from 40 400 results in increase the FKEA metrics.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper discusses and further expands the ideas of generative model entropic diversity evaluation presented in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the paper we discuss the limitations of the reference-free metrics (RKE, VENDI) approximation via the Random Fourier Features within Fourier-based Kernel Entropy Approximation (FKEA) method.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
30

---

Answer: [Yes]
Justification: The paper provides preliminary information, assumptions and definitions of the theoretical results along with relevant proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper outlines all necessary parameters experimental settings for reproducibility purposes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The submission comes along with demo code generate clusters and compute the scores.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic//uudees/Codeebbbmsss www.ssiiinooooiyoo for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/ccppblccgcccccoieeeec
wwweeeeeeoooSSeeese..eeaa ssissssssiypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Each experimental result accompanied by relevant parameters that aid in understanding of presented results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported due to limited amount of ground truth samples in the datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper states the computational resources used in the experimental setups.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurrps.... http:////wwnppp.cccppectee/////e http:///b../e.e.cccciceeeeeeeee.
Answer: [Yes]
Justification: The paper closely follows the code of ethics and all generated/doomlladee data has through sanity checks.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper discuss computational complexity and diversity evaluation of existing metrics (RKE, VENDI) and does not directly impact training of models.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
33

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: In this paper we discuss the evaluation of existing datasets and generative models. We do not release any custom datasets the public.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All relevant datasets and models are referenced in the paper and supplemental materials.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
34

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssii paperswi thcode. www.coo///aaaa com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve human evaluation and crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve human evaluation and crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "Fc8bCVO8Ok", "reviewer": "Reviewer_NG2d", "summary": "This study proposes a computationally efficient metric for evaluating the performance of recent generative models. It highlights the limitations of using reference data, which can restrict the applicability of evaluation methodologies, and instead suggests a method utilizing kernel functions without references.\nExperiments demonstrate that the proposed metric shows similar trends to existing metrics across various modalities and datasets, and qualitatively reflects the characteristics of the data in the metric computation process.", "questions": "- Please provide the extent of the speed improvement over existing method in actual experiments.\n- Provide experimental results based on the capabilities of different embedding models (e.g., compare performance measured using BERT for text with that using text-embedding-3-large).\n- Measure the correlation between the existing metrics and the proposed metric in your experiments.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Compared to existing metrics, the proposed method allows for quick metric computation through relatively simple calculations.\n* The calculation process reveals that the eigenvectors generated by the metric can semantically distinguish the generated outputs.\n* It is applicable using various embedding models.", "weaknesses": "* Compared to existing metrics, the speed improvement in performance measurement is not experimentally specified.\n* The numerical alignment with existing metrics is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer NG2d for his/her time and constructive feedback and suggestions on our work. The following is our response to the reviewer’s comments and questions.\n\n**1. FKEA computational costs compared to VENDI/RKE**\n\nRe: To address the reviewer’s comment, we have measured the time taken by FKEA-based and non-FKEA-based (via PyTorch eigendecomposition) computation of Vendi and RKE scores. Table 1 in the Rebuttal PDF summarizes the time complexity of FKEA evaluation compared to the baseline eigendecomposition-computed scores on one NVIDIA GeForce RTX 3090 GPU. While the baseline computation of RKE/VENDI were unaffordably expensive in terms of memory and compute power for sample sizes above 30k, FKEA-based entropy computation remained feasible at much larger sample sizes, e.g. 250k. We will add the table to the revised paper.\n\n\n**2. Effect of Embeddings on FKEA entropy evaluation** \n\nRe: Based on this comment, we performed numerical experiments and compared the Vendi/RKE diversity scores across various embeddings, including DinoV2, CLIP, SwAV, InceptionV3 for images and text-embedding-3-large/small, BERT and Gemini for text data. Figures 2a and 2b (in Rebuttal PDF) illustrate the numerical results. The results indicate that the proposed FKEA method could approximate the Vendi/RKE metrics across different embedding spaces.\n\n**3. Correlation between FKEA-approximated scores  and other diversity metrics**\n\nRe:  To address this point, we refer to Table 1 in the main text that compares FKEA-Vendi and FKEA-RKE with standard assessment metrics for diversity across several image-based generative models. The results indicate that FKEA-evaluated entropy scores correlate with existing metrics e.g. Recall and Coverage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n7LhqfMEru", "reviewer": "Reviewer_ohKy", "summary": "The work introduces a new method called Fourier-based Kernel Entropy Approximation (FKEA) to evaluate the diversity of data generated by generative models. Traditional evaluation metrics for generative models often rely on reference datasets, which may not always be available or suitable. Recently, reference-free entropy scores like VENDI and RKE have been proposed but suffer from high computational costs, especially with large-scale models.\n\nFKEA addresses this issue by leveraging the random Fourier features framework to reduce computational complexity. It approximates the eigenspectrum of the kernel matrix to estimate entropy scores efficiently. The method utilizes proxy eigenvectors derived from FKEA to identify modes in the diversity assessment of generated samples.", "questions": "see weaknesses section.", "limitations": "see weaknesses section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Paper is well-written and easy to follow. \n2. The contribution seems useful to the community.\n3. Thorough quantitative evaluation is present on text, image and video datasets. \n4. The paper mentions limitations and scope for improvement.", "weaknesses": "1. Evaluation seems to be limited to simple datasets. How would the method perform on complex image and video datasets? \n2. The reported metrics cover the basics of establishing the advantage of this method. Any other metrics that can be used to establish clear dominance of this method over existing methods?\n3. Qualitative results are hard to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer ohKy for his/her time and constructive feedback on our work. The following is our response to the reviewer’s comments and questions.\n\n**1. Datasets in the numerical evaluation** \n\nIn the main text, we have discussed the numerical results on the following benchmark datasets: \n- Images: MNIST, ImageNet, StyleGAN3 generated FFHQ\n- Text: Synthetic countries and landmarks dataset\n- Video: UCF101\n\nDue to the 9-page space limit, we had to defer our numerical results for the other datasets to the Appendix. We would like to refer the reviewer to sections B1, B4, B5 in the Appendix where we report our numerical results on several more large-scale datasets including FFHQ, AFHQ, MSCOCO, F-MNIST, CNN/Dailymail 3.0, Wikipedia, CMU Movie Corpus and Kinetics-400.\n\n\n**2. Comparison to other evaluation metrics in the literature**\n\nRe: We would like to clarify that our main contribution is to provide a computationally efficient method for approximating the existing reference-free Vendi and RKE evaluation scores. The Vendi and RKE metrics have been already proposed and analyzed in the literature (References [1-3]); however, they could be expensive to compute in practice. In this work, we leverage the framework of random Fourier features to reduce the computational expenses of evaluating generative models with RKE and Vendi metrics. The merits of these two metrics have been already studied in References [1-3], and as discussed in the introduction, their main application is in evaluation settings where no reference datasets are available for assessment. We refer the reviewer to Figure 1 in the Rebuttal PDF providing a toy example where the reference-based metrics did not perform satisfactorily, while the reference-free FKEA-Vendi and FKEA-RKE resulted in the proper ranking of the generative models.\n\n\n\n[1] Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. In Transactions on Machine Learning Research, 2023.\n\n[2] Mohammad Jalali, Cheuk Ting Li, and Farzan Farnia. An information-theoretic evaluation of generative models in learning multi-modal distributions. In Advances in Neural Information Processing Systems 2023.\n\n[3] Amey Pasarkar and Adji Bousso Dieng. Cousins of the vendi score: A family of similarity-based diversity metrics for science and machine learning. In International Conference on Artificial Intelligence and Statistics 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K387L2Wl4x", "reviewer": "Reviewer_ymdX", "summary": "This paper introduces the Fourier-based Kernel Entropy Approximation (FKEA) metric, which efficiently evaluates the diversity of generated samples. The key contributions of this work are twofold: (1) Compared to existing diversity metrics such as VENDI and RKE, the proposed metrics (i.e, FKEA-VENDI and FKEA-RKE) are more computationally efficient, reducing complexity to $O(n)$ for a sample size $n$. (2) The proposed metric is reference-free and can be used to assess the performance of large-scale image, text, and video datasets for generative models. Extensive experimental results demonstrate the effectiveness of the proposed metric.", "questions": "1. The proposed metrics require stochastic approximations on expectations (i.e., $\\mathbf{x}$ and $\\boldsymbol{\\omega}$). Could the authors report the variances of the evaluation results? Additionally, can the authors provide guidelines for choosing a sufficient number of samples?\n2. Is there any underlying intuition for selecting $\\sigma$ and $r$? Will an inappropriate selection of $\\sigma$ and $r$ lead to an inaccurate assessment of the true generative quality?\n3. Could the authors provide experimental results to demonstrate that reference-dependent metrics (e.g., Recall and Coverage) may fail to measure sample quality, while reference-free metrics may be successful?\n4. Could the authors provide an evaluation time comparison between the calculation of FKEA-based metrics and their original metrics? For example, an evaluation time comparison between FKEA-VENDI and VENDI would be helpful.", "limitations": "The authors have allocated a section (i.e., Line 292) for the discussion of potential limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper writing has a clear structure and the theoretical results are technically sound.\n- The proposed metric exhibits improved theoretical computational efficiency (i.e., Line 13) and can be calculated without sophisticated hardware setups (Line 245).\n- The method can be broadly applied to assess the performance of various generative models for images, texts, and videos.", "weaknesses": "- The proposed evaluation method relies on a number of parameters, e.g., $\\sigma$ and $r$. The values of these parameters seem to vary significantly according to different tasks. For example, the authors adopt $2r=4,000$ and $\\sigma=7$ for the experiment on the MNIST dataset, and $2r=16,000$ and $\\sigma=25$ for the experiment on the ImageNet dataset. How these parameters influence the evaluation results remains unexplored. (See Questions 1 and 2)\n- The paper could be improved by incorporating a toy example to illustrate the difference between reference-dependent and reference-free metrics. The discussions in Sections 1 and 2 are not concrete enough. (See Question 3)\n- One of the key contributions of this paper is the theoretical improvements in the computational complexities of the FKEA-based metrics. An evaluation time comparison between FKEA-based metrics (e.g., FKEA-VENDI and FKEA-RKE) and their original metrics (e.g., VENDI and RKE) is missing. (See Question 4)\n\n---\n\n**Minor Point**\n\n- There is a typo (i.e., $x \\to \\mathbf{x}$) in Line 157.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer ymdX for his/her time and constructive feedback and suggestions on our work. The following is our response to the reviewer’s comments and questions.\n\n**1. Variance of estimating entropy scores using FKEA**\n\nRe: To address the reviewer’s question, we measured the standard deviation of estimated FKEA entropy scores over 10 trials of running the experiment. We performed the variance estimation three times under: 1) independently drawn random Fourier features and fixed generated samples (across trials), 2) fixed (across trials) random Fourier features $\\omega$ and independently drawn generated samples, 3) independently drawn random Fourier features and generated samples.\n\nThe estimated entropy scores and their standard deviation are presented in Table 2 of the Rebuttal PDF. The results indicate that FKEA with $r=4000$ and $r=8000$ can yield approximations with relatively low variance.\n\n\n**2. Selection of bandwidth $\\sigma$ and Fourier feature size $r$ hyperparameters**\n\nRe: We would like to clarify that the kernel bandwidth hyperparameter $\\sigma$ is a specification of kernel function which is separated from the FKEA method designed to approximate the Vendi [1,3] or RKE [2] scores with the evaluator’s chosen kernel function. Therefore, in our experiments we chose the $\\sigma$ parameter following References [1-3] that originally proposed the scores. Also, note that, based on the theoretical results in [2], a proper choice of hyperparameter $\\sigma$ should be greater than the maximum standard deviation of the modes of a mixture distribution and be smaller than the minimum Euclidean distance between the means of different modes.\n\nOn the other hand, the number of Fourier features $r$ is a hyperparameter of the FKEA method. Based on Theorem $1$, the selection $r=\\frac{8(\\log(n) + \\log(1/2\\delta))}{\\epsilon^2}$ will be sufficient to guarantee an $\\epsilon$-bounded approximation error for the kernel matrix’s eigenvalues with probability $1-\\delta$. In our experiments, we observed that the choice of $r=8000$ for standard image datasets results in an estimation whose standard deviation is bounded by 1% of the estimated entropy.   \n\n\n**3. Merits of reference-free metrics** \n\nRe: As discussed in the introduction, reference-free metrics are highly useful when the evaluator does not have access to a proper reference dataset. This can happen in scenarios where generative models produce outputs that are not represented in the standard datasets. To illustrate this, we included a toy example in Figure 1 (Rebuttal PDF) using a synthetic dataset of elephants generated by the Stable-Diffusion-XL text-to-image model. When the dataset consists of regular elephants, a reference set of “elephant” samples of ImageNet provides an accurate assessment of diversity. However, when the model is prompted to generate elephants with unrealistic colors, the internal diversity increases—something that ImageNet-based reference-dependent evaluations could not capture.\n\nIn this scenario, reference-dependent metrics seemed to report either lower score (Coverage) or no significant change (Recall). In contrast, the FKEA-computed Vendi and RKE scores could identify the increased diversity, suggesting its effectiveness in evaluating generative models in contexts where traditional datasets do not fully capture the generated data's diversity. We will include this example in the revised text. \n\n\n**4. Computational costs of FKEA** \n\nRe: Table 1 in the included figures (see attached PDF to global response) summarises the improved time complexity compared to Vendi/RKE scores computed by standard PyTorch eigendecomposition method using one NVIDIA GeForce RTX 3090 GPU. The FKEA method could significantly reduce the compute and memory costs, while providing an accurate estimation of the original metric with low variance (Table 2 in Rebuttal PDF). Moreover, RKE/VENDI were too expensive to compute with moderately high sample size e.g. 40k; however, FKEA remained applicable with much higher sample sizes such as 250k.\n\n[1] Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. In Transactions on Machine Learning Research, 2023.\n\n[2] Mohammad Jalali, Cheuk Ting Li, and Farzan Farnia. An information-theoretic evaluation of generative models in learning multi-modal distributions. In Advances in Neural Information Processing Systems 2023.\n\n[3] Amey Pasarkar and Adji Bousso Dieng. Cousins of the vendi score: A family of similarity-based diversity metrics for science and machine learning. In International Conference on Artificial Intelligence and Statistics 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive feedback and suggestions. We have responded to each reviewer's comments and questions under the review-box. Here we upload the 1-page PDF including the figures and plots discussed in our responses.

Meta-review: The authors proposed an algorithm to evaluate the diversity of generated data in generative modelling. Most metrics used in applications rely on reference datasets and reference-free entropy scores VENDI and RKE are too computationally intensive in high dimensional settings. In this paper, the authors use the random Fourier features framework to propose an approximation of these scores reducing significantly the computational complexity. 

The paper was reviewed by three experts who all gave a borderline accept score highlighting in particular that the paper is well written and that the contribution is useful for the community. The main limitation were related to the experiments to highlight the claim of the authors. 

Providing efficient ways to evaluate generative models is a crucial challenge in particular for practitioners as these models are at the heart of a very intense research activity. The authors provide a very comprehensive procedure to evaluate the diversity of generated samples based on  random Fourier features supported by theoretical guarantees and extensive simulations using toy data, image and video data. The additional experiments run during the rebuttal improve the paper and confirm the reduced complexity which makes the proposed approach interesting in large scale settings where  original entropy-based diversity evaluation scores are not available.
Decision: Accept (poster)