Towards a Scalable Reference-Free Evaluation of Generative Models

Authors: azim  ospanov
jingwei  zhang
mohammad  jalali
xuenan  cao
andrej  bogdanov
farzan  farnia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 06:47:06

Abstract: While standard evaluation scores for generative models are mostly reference-based, a reference-dependent assessment of generative models could be generally difficult due to the unavailability of applicable reference datasets. Recently, the reference-free entropy scores, VENDI and RKE, have been proposed to evaluate the diversity of generated data. However, estimating these scores from data leads to significant computational costs for large-scale generative models. In this work, we leverage the random Fourier features framework to reduce the metrics' complexity and propose the *Fourier-based Kernel Entropy Approximation (FKEA)* method. We utilize FKEA's approximated eigenspectrum of the kernel matrix to efficiently estimate the mentioned entropy scores. Furthermore, we show the application of FKEA's proxy eigenvectors to reveal the method's identified modes in evaluating the diversity of produced samples. We provide a stochastic implementation of the FKEA assessment algorithm with a complexity $O(n)$ linearly growing with sample size $n$. We extensively evaluate FKEA's numerical performance in application to standard image, text, and video datasets. Our empirical results indicate the method's scalability and interpretability applied to large-scale generative models.  The codebase is available at [https://github.com/aziksh-ospanov/FKEA](https://github.com/aziksh-ospanov/FKEA).

Main paper: 
 introduction: 
 A quantitative comparison of generative models requires evaluation metrics to measure the quality and diversity of the models' produced data. Since the introduction of variational autoencoders (VAEs) [3] , generative adversarial networks (GANs) [4] , and diffusion models [5] that led to impressive empirical results in the last decade, several evaluation scores have been proposed to assess generative models learned by different training methods and architectures. Due to the key role of evaluation criteria in comparing generative models, they have been extensively studied in the literature.While various statistical methods have been applied to measure the fidelity and variety of a generative model's produced data, the standard scores commonly perform a reference-based evaluation of generative models, i.e., they quantify the characteristics of generated samples in comparison to a reference distribution. The reference distribution is usually chosen to be either the distribution of samples in the test data partition or a comprehensive dataset containing a significant fraction of real-world sample types, e.g. ImageNet [6] for evaluating image-based generative models.To provide well-known examples of reference-dependent metrics, note that the distance scores, Fréchet Inception Distance (FID) [7] and Kernel Inception Distance (KID) [8] , are explicitly referencebased, measuring the distance between the generative and reference distributions. Similarly, the standard quality/diversity score pairs, Precision/Recall [9, 10] and Density/Coverage [11] , perform the evaluation in comparison to a reference dataset. Even the seemingly reference-free Inception Score (IS) [12] can be viewed as implicitly reference-based, since it quantifies the variety and fidelity of data based on the labels and confidence scores assigned by an ImageNet pre-trained neural net, where ImageNet implicitly plays the role of the reference dataset. The reference-based nature of these evaluation scores is desired in many instances including standard image-based generative models, where either a sufficiently large test set or a comprehensive reference dataset such as ImageNet is available for the reference-based evaluation.On the other hand, a reference-based assessment of generative models may not always be feasible, because the selection of a reference distribution may be challenging in a general learning scenario. For example, in prompt-based generative models where the data are created in response to a user's input text prompts, the generated data could follow an a priori unknown distribution depending on the specific distribution of the user's input prompts. Figure 1 shows one such example where we compare reference-based diversity scores of regular and colored elephant image samples generated by Stable Diffusion XL [13] . While the diversity of the colored images looks significantly higher to the human eye, the evaluated reference-based FID, Recall, and Coverage metrics do not suggest a higher diversity. As this example suggests, a proper reference-based evaluation of every user's generated data would require a distinct reference dataset, which may not be available to the user during the assessment time. Moreover, finding a comprehensive text or video dataset to choose as the reference set would be more difficult compared to image data, because the higher length of text and video samples could significantly contribute to their variety, requiring an inefficiently large reference set to cover all text or video sample types.The discussed challenging scenarios of conducting a reference-based evaluation highlight the need for reference-free assessment methods that remain functional in the absence of a reference dataset. Recently, entropy-based diversity evaluation scores, the VENDI metric family [1, 14] and RKE score [2] , have been proposed to address the need for reference-free assessment metrics. These scores calculate the entropy of the eigenvalues of a kernel similarity matrix for the generated data. Based on the theoretical results in [2] , the evaluation process of these scores can be interpreted as an unsupervised identification of the generative model's produced sample clusters, followed by the entropy calculation for the frequencies of the detected clusters. In Figure 1 , we observe that the reference-free VENDI and RKE scores grow when the generated samples are colored, which is due to the increase in the quantity of identified clusters in the colored case.While the VENDI and RKE entropy scores provide reference-free assessments of generative models, estimating these scores from generated data could incur significant computational costs. In this work, we show that computing the precise RKE and VENDI scores would require at least Ω(n 2 ) and Ω(n 2.373 ) 4 computations for a sample size n, respectively. While the randomized projection methods in [15, 1] can reduce the computational costs to O(n 2 ) for a general VENDI α score, the quadratic growth would be a barrier to the method's application to large n values. Although the computational expenses could be reduced by limiting the sample size n, an insufficient sample size would lead to significant error in estimating the entropy scores. As an example on the ImageNet dataset, Figure 7 in the Appendix shows the adverse effects of limiting the sample size on the quality of clusters used in the calculation of the VENDI scores.To overcome the challenges of computing the scores, we leverage the random Fourier features (RFFs) framework [16] and develop a scalable entropy-based evaluation method that can be efficiently applied to large sample sizes. Our proposed method, Fourier-based Kernel Entropy Approximation (FKEA), is designed to approximate the kernel covariance matrix using the RFFs drawn from the Fourier transform-inverse of a target shift-invariant kernel. We prove that using a Fourier feature size r = O log n ϵ 2, FKEA computes the eigenspace of the kernel matrix within an ϵ-bounded error. Furthermore, we demonstrate the application of the eigenvectors of the FKEA's proxy kernel matrix for identifying the sample clusters used in the reference-free evaluation of entropic diversity. Finally, we present numerical results of the entropy-based evaluation of standard generative models using the baseline eigendecomposition and our proposed FKEA methods. In our experiments, the baseline spectral decomposition algorithm could not efficiently scale to sample sizes above a few ten thousand. On the other hand, our stochastic implementation of the FKEA method could scalably apply to large sample sizes. Utilizing the standard embeddings of image, text, and video data, we tested the FKEA assessment while computing the sample clusters and their frequencies in application to large-scale datasets and generative models. Here is a summary of our work's main contributions:• Characterizing the computational complexity of the kernel entropy scores of generative models, • Developing the Fourier-based FKEA method to approximate the kernel covariance eigenspace and entropy of generated data, • Proving guarantees on FKEA's required size of random Fourier features indicating a complexity logarithmically growing with the dataset size, • Providing numerical results on FKEA's reference-free assessment of large-scale image,text, videobased datasets and generative models. 
related work: 
 Evaluation of deep generative models. The assessment of generative models has been widely studied in the literature. The existing scores either quantify a distance between the distributions of real and generated data, as in FID [7] and KID [8] scores, or attempt to measure the quality and diversity of the trained generative models, including the Inception Score [12] , quality/diversity metric pairs Precision/Recall [9, 10] and Density/Coverage [11] . The mentioned scores are referencebased, while in this work we focus on reference-free metrics. Also, we note that the evaluation of memorization and novelty has received great attention, and several scores including the authenticity score [17] , the feature likelihood divergence [18] , and the rarity score [19] have been proposed to quantify the generalizability and novelty of generated samples. Note that the evaluation of novelty and generalization is, by nature, reference-based. On the other hand, our study focuses on the diversity of data which can be evaluated in a reference-free way as discussed in [1, 2] .Role of embedding in quantitative evaluation. Following the discussion in [20] , we utilize DinoV2 [21] image embeddings in most of our image experiments, as [20] 's results indicate DinoV2 can yield scores more aligned with the human notion of diversity. As noted in [22] , it is possible to utilize other non-ImageNet feature spaces such as CLIP [23] and SwAV [24] as opposed to InceptionV3 [25] to further improve metrics such as FID. In this work, we mainly focus on DinoV2 feature space, while we note that other feature spaces are also compatible with entropy-based diversity evaluation.Diversity assessment for text-based models. To quantify the diversity of text data, the n-gram-based methods are commonly used in the literature. A well-known metric is the BLEU score [26] , which is based on the geometric average of n-gram precision scores times the Brevity Penalty. To adapt BLEU score to measure text diversity, [27] proposes the Self-BLEU score, calculating the average BLEU score of various generated samples. To further isolate and measure diversity, N-Gram Diversity scores [28, 29, 30] were proposed and defined by a ratio between the number of unique n-grams and overall number of n-grams in the text. Other prominent metrics include Homogenization (ROUGE-L) [31] , FBD [32] and Compression Ratios [33] .Kernel PCA, Spectral Cluttering, and Random Fourier Features. Kernel PCA [34] is a wellstudied method of dimensionality reduction that utilizes the eigendecomposition of the kernel matrix, similar to the kernel-based diversity evaluation methods in [1, 2] . The related papers [35, 36] study the connections between kernel PCA and spectral clustering. Also, the analysis of random Fourier features [16] for performing scalable kernel PCA has been studied in [37, 38, 39, 40, 41] . We note that while the mentioned works characterize the complexity of estimating the eigenvectors, our analysis focuses on the complexity of computing the kernel matrix's eigenvalues via Fourier features, as we primarily seek to quantify the diversity of generated data using the kernel matrix's eigenvalues. 
preliminaries: 
 Consider a generative model G generating random samples x 1 , . . . , x n ∈ R d following the model's probability distribution P G . In our analysis, we assume the n generated samples are independently drawn from P G . Note that in VAEs [3] and GANs [4] , the generative model G is a deterministic function G : R r → R d mapping an r-dimensional latent random vector Z ∼ P Z from a known distribution P Z to G(Z) distributed according to P G . On the other hand, in diffusion models, G represents an iterative random process that generates a sample from P G . The goal of a sample-based diversity evaluation of generative model G is to quantify the variety of its generated data x 1 , . . . , x n .Following standard definitions, k : R d ×R d → R is called a kernel function if for every integer n ∈ N and set of inputs x 1 , . . . , x n ∈ R d , the kernel similarity matrix K = k(x i , x j ) n×n is positive semi-definite. We call a kernel function k normalized if for every input x we have k(x, x) = 1. A well-known example of a normalized kernel function is the Gaussian kernel k Gaussian(σ 2 ) with bandwidth parameter σ 2 defined ask Gaussian(σ 2 ) x, x ′ := exp - x -x ′ 2 2 2σ 2For every kernel function k, there exists a feature map ϕ : R d → R m such that k(x, x ′ ) = ⟨ϕ(x), ϕ(x ′ )⟩ is the inner product of the m-dimensional feature maps ϕ(x) and ϕ(x ′ ). Given a kernel k with feature map ϕ, we define the kernel covariance matrix C X ∈ R m×m of a distribution P X asC X := E X∼P X ϕ(X)ϕ(X) ⊤ = p X (x)ϕ(x)ϕ(x) ⊤ dxThe above matrix C X is positive semi-definite with non-negative values. Furthermore, assuming a normalized kernel k, it can be seen that the eigenvalues of C X will add up to 1 (i.e., it has unit trace Tr(C X ) = 1), providing a probability model. Therefore, one can consider the entropy of C X 's eigenvalues as a quantification of the diversity of distribution P X based on the kernel similarity score k. Here, we review the general family of Rényi entropy used to define VENDI and RKE scores.Definition 1. For a positive semi-definite matrix C X ∈ R m×m with eigenvalues λ 1 , . . . , λ m , the order-α Rényi entropy H α (C X ) for α > 0 is defined asH α (C X ) := 1 1 -α log m i=1 λ α iTo estimate the entropy scores from finite empirical samples x 1 , . . . , x n , we consider the empirical kernel covariance matrix C X defined as C X := 1 n n i=1 ϕ x i ϕ x i ⊤ . This matrix provides an empirical estimation of the population kernel covariance matrix C X .It can be seen that the m × m empirical matrix C X and normalized kernel matrix 1 n K = 1 n k(x i , x j ) n×n share the same non-zero eigenvalues. Therefore, to compute the matrix-based entropy of the empirical covariance matrix C X , one can equivalently compute the entropy of the eigenvalues of the kernel similarity matrix K. This approach results in the definition of the VENDI and RKE diversity scores: [1] defines the family of VENDI scores asVENDI α (x 1 , . . . , x n ) := exp H α 1 n K = n i=1 λ α i 1 1-α ,where λ 1 , . . . , λ n denote the eigenvalues of the kernel matrix 1 n K. Also, [2] proposes the RKE score, which is the special order-2 Renyi entropy, RKE(x 1 , . . . ,x n ) = exp(H 2 ( 1 n K)).To compute RKE without computing the eigenvalues, [2] points out the RKE score reduces to the Frobenius norm ∥ • ∥ F of the kernel matrix as follows:RKE(x 1 , . . . , x n ) = 1 n K -2 F = 1 n 2 n i=1 n j=1 k x i , x j 2 -1A kernel function k is called shift-invariant, if there exists a function κ :R d → R such that k(x, x ′ ) = κ(x -x ′ ) for every x, x ′ ∈ R d .Bochner's theorem proves that a function κ : R d → R will lead to a shift-invariant kernel similarity score κ(x -x ′ ) between x, x ′ if and only if its Fourier transform κ : R d → R is non-negative everywhere (i.e, κ(ω) ≥ 0 for every ω). Note that the Fourier transform κ is defined asκ(ω) := 1 (2π) d κ(x) exp -iω ⊤ x dxSpecifically, Bochner's theorem shows the Fourier transform κ of a normalized shift-invariant kernel k(x, x ′ ) = κ(x-x ′ ), where κ(0) = 1, will be a probability density function (PDF). The framework of random Fourier features (RFFs) [16] utilizes independent samples drawn from PDF κ to approximate the kernel function. Here, given independent samples ω 1 , . . . , ω r ∼ κ, we form the following proxy feature map ϕ r :EQUATIONAs demonstrated in [16, 42] , the 2r-dimensional proxy map ϕ r can approximate the kernel function as k(x,x ′ ) = E ω∼ κ cos(ω ⊤ x) cos(ω ⊤ x ′ ) + sin(ω ⊤ x) sin(ω ⊤ x ′ ) ≈ ϕ r (x) ⊤ ϕ r (x ′ ). 
computational complexity of vendi & rke scores: 
 As discussed, computing RKE and general VENDI α scores requires computing the order-α entropy of kernel matrix 1 n K. Using the standard definition of α-norm ∥v∥ α = n i=1 |v i | α 1/α , we observe that the computation of VENDI α score is equivalent to computing the α-norm ∥λ∥ α of the n-dimensional eigenvalue vector λ = [λ 1 , . . . , λ n ] where λ 1 ≥ • • • ≥ λ n are the sorted eigenvalues of the normalized kernel matrix 1 n K. In the following theorem, we prove that except order α = 2, which is the RKE score, computing any other VENDI α score is at least as expensive as computing the product of two n × n matrices. Therefore, the theorem suggests that the computational complexity of every member of the VENDI family is lower-bounded by Ω(n 2.372 ) which is the least known cost of multiplying n × n matrices.In the theorem, we suppose B is any fixed set of "basis" functions. A circuit C is a directed acyclic graph each of whose internal nodes is labeled by a gate coming from a set B. A subset of gates are designated as outputs of C. A circuit with n source nodes and m outputs computes a function from R n to R m by evaluating the gate at each internal gate in topological order. The size of a circuit is the number of gates. Also, ∇B is the basis consisting of the gradients of all functions in B. We will provide the proof of the theorems in the Appendix. Theorem 1. If VENDI α (K) for α ̸ = 2 is computable by a circuit C of size s(n) over basis B, then n × n matrices can be multiplied by a circuit C of size O(s(n)) over basis B ∪ ∇B ∪ {+, ×}. Remark 1. The smallest known circuits for multiplying n × n matrices have size Θ(n ω ), where ω ≈ 2.372. Despite tremendous research efforts only minor improvements have been obtained in recent years. There is evidence that ω is bounded away from 2 for certain classes of circuits [43, 44] . In contrast, S 2 is computable in quadratic time Θ(n 2 ) in the basis B = {×, +, log}.The above discussion indicates that except the RKE(x 1 , . . . , x n ), i.e. order-2 Renyi entropy, whose computational complexity is quadratically growing with sample size Θ(n 2 ), the other members of the VENDI family VENDI α would have a super-quadratic complexity on the order of O(n 2.372 ). In practice, the computation of VENDI α scores is performed by the eigendecomposition of the n × n kernel matrix that requires O(n 3 ) computations for precise computation and O(n 2 M ) computations using a randomized projection onto an M -dimensional space [15, 1] . 
a scalable fourier-based method for computing kernel entropy scores: 
 As we showed earlier, the complexity of computing RKE and VENDI scores are at least quadratically growing with the sample size n. The super-linear growth of the scores' complexity with sample size n can hinder their application to large-scale datasets and generative models with potentially hundreds of sample types. In such cases, a proper entropy estimation should be performed over potentially hundreds of thousands of data, where the quadratic complexity of the scores would be a significant barrier toward their accurate estimation.Here, we consider a shift-invariant kernel matrix k(x, x ′ ) = κ(x -x ′ ) where κ(0) = 1 and propose applying the random Fourier features (RFF) framework [16] to perform an efficient approximation of the RKE and VENDI scores. To do this, we utilize the Fourier transform κ that, according to Bochner's theorem, is a valid PDF, and we independently generate ω 1 , . . . , ω r iid ∼ κ. Note that in the case of the Gaussian kernel k Gaussian(σ 2 ) , the corresponding PDF will be an isotropic Gaussian N (0, 1 σ 2 I d ) with zero mean and covariance matrix 1 σ 2 I d . Then, we consider the RFF proxy feature map ϕ r : R d → R 2r as defined in (1) and define the proxy kernel covariance matrix C X,r ∈ R 2r×2r :EQUATIONNote that the 2r × 2r matrix C X,r has the same non-zero eigenvalues as the n × n RFF proxy kernel matrix 1 n K r , and therefore can be utilized to approximate the eigenvalues of the original n × n kernel matrix 1 n K. Therefore, we propose the Fourier-based Kernel Entropy Approximation (FKEA) method to approximate the RKE and VENDI α scores as follows:EQUATIONFKEA-VENDI α x 1 , . . . , x n = exp H α ( C X,r ) = 2r i=1 λ α r,i 1 1-α (4)Note that in the above, λ α r,i denotes the ith eigenvalue of the 2r × 2r matrix C X,r . We remark that the computation of both FKEA-RKE and FKEA-VENDI α can be done by a stochastic algorithm which computes the proxy covariance matrix (2) by summing the sample-based 2r × 2r matrix terms, and then computing the resulting matrix's Frobenius norm for RKE score or all the 2r matrix's eigenvalues for a general VENDI α with α ̸ = 2. Algorithm 1 presents the steps of the FKEA method where the computation needed for the proxy kernel covariance matrix is O(n) and grows only linearly with sample size n.Therefore, to show the FKEA method's scalability, we need to bound the required RFF size 2r for an accurate approximation of the original n × n kernel matrix. The following theorem proves that the needed feature size will be O log n ϵ 2 for an ϵ-accurate approximations of the matrix's eigenspace. Compute the RFF feature for x i :ϕ r (x i ) = 1 √ r cos(ω ⊤ 1 x i ), sin(ω ⊤ 1 x i ), • • • , cos(ω ⊤ r x i ), sin(ω ⊤ r x i ) ⊤ 7:Update C:C ← C + 1 n ϕ r (x i ) ϕ r (x i ) ⊤8: end for 9: Perform eigendecomposition on the covariance matrix:{ λ 1 , . . . , λ 2r } ← Eigendecomposition( C) 10:Compute VENDI and RKE metrics using the eigenvalues λ 1 , . . . , λ 2rTheorem 2. Consider a shift-invariant kernel k(x, x ′ ) = κ(x -x ′ ) where κ(0) = 1. Suppose ω 1 , . . . , ω r ∼ κ are independently drawn from PDF κ. Let λ 1 ≥ . . . ≥ λ n be the sorted eigenvalues of the normalized kernel matrix 1 n K = 1 n k(x i , x j ) n×n . Also, consider the eigenvalues of λ 1 ≥ . . . ≥ λ 2r of random matrix C X,r with their corresponding eigenvectors v 1 , . . . , v 2r . Let λ j = 0 for every j > 2r. Then, for every δ > 0, the following holds with probability at least 1 -δ:n i=1 λ i -λ i 2 ≤ 8 log(n/2δ) r and n i=1 1 n K v i -λ i v i 2 2 ≤ 32 log(n/2δ) r ,wherev i := r j=1 sin v ⊤ 2j x i v 2j + cos v ⊤ 2j-1 x i v 2j-1is the ith proxy eigenvector for 1 n K. Corollary 1. In the setting of Theorem 2, the following approximation guarantees hold for RKE and VENDI α scores• For every VENDI α with α ≥ 2, including RKE for α = 2, the following dimension-independent bound holds with probability at least 1 -δ:FKEA-VENDI α x 1 , . . . , x n 1-α α -VENDI α x 1 , . . . , x n 1-α α ≤ 8 log(n/2δ) r• For every VENDI α with 1 ≤ α < 2, assuming a finite dimension for the kernel feature map dim(ϕ) = m, the following bound holds with probability at least 1 -δ:FKEA-VENDI α x 1 , . . . , x n 1-α α -VENDI α x 1 , . . . , x n 1-α α ≤ m 1 α -1 2 8 log(n/2δ) r Remark 2.According to the theoretical results in [45] , the top-t eigenvectors of kernel covariance matrix C X will correspond to the mean of the modes of a mixture distribution with t well-separable modes. Theorem 2 shows for every 1 ≤ i ≤ 2r, FKEA provides the proxy score function u i : R d → R that assigns a likelihood score for an input x to belong to the ith identified mode:EQUATIONTherefore, one can compute the above FKEA-based score for each of the 2r eigenvectors over a sample set, and use the samples with the highest scores according to every u i to characterize the i sample cluster captured by the FKEA method.  
numerical results: 
 We evaluated the FKEA method on several image, text, and video datasets to assess its performance in quantifying diversity in different domains. In the experiments, we computed the empirical covariance matrix of 2r-dimensional Fourier features with a Gaussian kernel with bandwidth parameter σ tuned for each dataset, and then applied FKEA approximation for the VENDI 1 , VENDI 1.5 , and the RKE (same as VENDI 2 ) scores. An algorithm to compute these scores is presented in Algorithm 1.Experiments were conducted on RTX3090 GPUs. We interpreted the modes identified by FKEA entropy-based diversity evaluation using the eigenvectors of the proxy covariance matrix as discussed in Remark 2. For each eigenvector, we presented the training data with maximum eigenfunction values corresponding to the eigenvector.Time Complexity of FKEA metrics. To highlight the computational advantages of transitioning to FKEA, Table 1 presents a comparison of the metric computations for VENDI and RKE on the ImageNet dataset, with sample sizes ranging from 10k to 250k. Our results show that VENDI and RKE become computationally intractable due to memory overflow. In contrast, the FKEA method efficiently scales up to n = 250k samples, maintaining optimal computational time.Experimental Results on Image Data. To investigate the FKEA method's diversity evaluation in settings where we know the ground-truth clusters and their quantity, we simulated an experiment on the colored MNIST [46] data with the images of 10 colored digits as shown in Figure 2 . We evaluated the FKEA approximations of the diversity scores while including samples from t digits for t ∈ {1, . . . , 10}. The plots in Figure 2 indicate the increasing trend of the scores and FKEA's tight We conducted an experiment on the ImageNet data to monitor the scores' behavior evaluated for 50k samples from an increasing number of ImageNet labels. Figure 3 shows the increasing trend of the scores as well as the top-9 samples representing the top-4 identified clusters used for the entropy calculation. Also, Figure 4 presents the FKEA approximated entropy scores with different truncation factors in StyleGAN3 [47] on 30k generated data for each truncation factor, showing the increasing diversity scores with the truncation factor. We defer discussing the results on AFHQ [48] , MS-COCO [49] , F-MNIST [50] datasets to the Appendix. Experimental Results on Text and Video Data. To perform experiments on the text data with known clustering ground-truth, we generated 500,000 paragraphs using GPT-3.5 [51] about 100 randomly selected countries (5k samples per country). In the experiments, the text embedding used was text-embedding-3-large [52, 53, 51] . We evaluated the diversity scores over data subsets of size 50k with different numbers of mentioned countries. Figure 5 shows the growing trend of the diversity scores when including more countries. The figure also shows the countries mentioned in the top-6 modes provided by FKEA-based principal eigenvectors, which shows the RFF-based clustering of countries correlates with their continent and geographical location. We discuss the numerical results on non-synthetic text datasets, Wikipedia, CNN/Dailymail [54] [55], CMU Movie Corpus [56] , in the Appendix.For video data experiments, we considered two standard video datasets, UCF101 [57] and Kinetics-400 [58] . Following the video evaluation literature [59, 60] , we used the I3D pre-trained model [61] as embedding, which maps a video sample to a 1024-dimensional vector. As shown in Figure 6 , increasing the number of video classes of test samples led to an increase in the FKEA approximated diversity metrics. Also, while the samples identified for the first identified cluster look broad, the next modes seemed more specific. We discuss the results of the Kinetics dataset in the Appendix.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Each experimental result is accompanied by relevant parameters that aid in understanding of presented results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this work, we proposed the Fourier-based FKEA method to efficiently approximate the kernel-based entropy scores VENDI α and RKE scores. The application of FKEA results in a scalable referencefree evaluation of generative models, which could be utilized in applications where no reference data is available for evaluation. A future direction to our work is to study the sample complexity of the matrix-based entropy scores and the FKEA's approximation under high-dimensional kernel feature maps, e.g. the Gaussian kernel. Also, analyzing the role of feature embedding in the method's application to text and video data would be interesting for future exploration.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not reported due to limited amount of ground truth samples in the datasets.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The paper discuss computational complexity and diversity evaluation of existing metrics (RKE, VENDI) and does not directly impact training of models. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: In this paper we discuss the evaluation of existing datasets and generative models. We do not release any custom datasets to the public.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All relevant datasets and models are referenced in the paper and supplemental materials.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve human evaluation and crowdsourcing.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve human evaluation and crowdsourcing.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Proof of Theorem 1The proof of Theorem 1 combines third ingredients. The first is the relation between the circuit size of a function C and of its partial derivatives ∇C = (∂C/∂x 1 , . . . , ∂C/∂x n ). Lemma 1. The function ∇C has a circuit over basis ∇B ∪ {+, ×} whose size is within a constant factor of the size of C.Lemma 1 is a feature of the backpropagation algorithm [62, 63] . This is a linear-time algorithm for constructing a circuit for ∇C given the circuit C as input. In contrast, the forward propagation algorithm allows efficient computation of a single (partial) derivative even for circuits with multivalued outputs, giving the second ingredient: Lemma 2. Let C be a circuit over basis B and t be an input to C. There exists a circuit that computes the derivative ∂g/∂t for every gate g of C over basis ∇B ∪ {+, ×} whose size is within a constant factor of the size of C.The last ingredient is the following identity. For a scalar function f over the complex numbers and matrix X diagonalizable as U ΛU T , we define f (X) to be the function U f (Λ)U T where f is applied entry-wise to the diagonal matrix Λ. Lemma 3. For every f that is analytic over an open domain Ω containing all sufficiently large complex numbers and every matrix X whose spectrum is contained in Ω, ∇Tr(f (X)) = f ′ (X).We first illustrate the proof in the special case when ∥X∥ is within the radius of convergence of f . Namely, f (x) is represented by the absolutely convergent seriesf (k) (0)x k /k! for all |x| ≤ ρ. Then f (X) = f (k) (0)X k /k! assuming ∥X∥ ≤ ρ.By linearity (and using the fact that derivatives preserve radius of convergence) it is sufficient to show thatEQUATIONwhich can be verified by explicit calculation: Both sides equal kX k-1 . This is sufficient to establish Theorem 1 for all integer α > 2.Proof of Lemma 3. The Cauchy integral formula for matrices yields the representationf (X) = 1 2πi C f (z)(zI -X) -1 dz,for any closed curve C whose interior contains the spectrum of X. As (zI -X) -1 is continuous along C, we can writeEQUATIONChoosing C to be a circle of radius ρ larger than the spectral norm of X, for all z of magnitude ρ we have the identity(zI -X) -1 = z -1 (I -z -1 X) -1 = z -1 ∞ k=0 z -k X kAs the series z -k ∇TrX k = kz -k X k-1 converges absolutely in spectral norm, using (6) we obtain the identity ∇Tr(zI -X) -1 = d(zI -X) -1 /dX, namely the lemma holds for the function f (X) = (zI -X) -1 . Plugging into (7) and exchanging the order of integration and derivation proves the lemma.Proof of Theorem 1. Assume Trρ α (resp., -Trρ log ρ) has circuit size s(d). By Lemma 1 and Lemma 3, ∇Trρ α = αρ α-1 (resp., -∇Trρ log ρ = log ρ -1/ ln 2) has circuit size O(s(d)). For every symmetric matrix X and sufficiently small t, the matrix ρ = I + tX is positive semi-definite. By Lemma 2 the R d 2 -valued function ∂ 2 ρ/∂t 2 has circuit size O(s d ). The value of this function at t = 0 is α(α -1)(α -2)X 2 (resp., X 2 ), namely the square of the input matrix X up to constant. Finally, computing the product AB reduces to squaring the symmetric matrix  A T B A B T   .Assuming that the shift-invariant kernel k(x, x ′ ) = κ(x -x ′ ) is normalized (i.e. κ(0) = 1), then the Fourier transform κ is a valid PDF according to Bochner's theorem and also an even function because κ takes real values. Then, we havek x, x ′ = κ σ x -x ′ (a) = κ σ (ω) exp iω ⊤ (x -x ′ ) dω (b) = κ σ (ω) cos ω ⊤ (x -x ′ ) dω = E ω∼ κ cos ω ⊤ (x -x ′ ) = E ω∼ κ cos ω ⊤ x) cos ω ⊤ x ′ ) + sin ω ⊤ x) sin ω ⊤ x ′ )Here, (a) comes from the synthesis property of the Fourier transform. (b) holds since κ σ is an even function, resulting in a zero imaginary term in the Fourier synthesis.Therefore, since cos ω ⊤ y ≤ 1 for all ω and y, one can apply Hoeffding's inequality to show for independently drawn ω 1 , . . . , ω r iid ∼ κ the following probably correct bound holds:EQUATIONTherefore, as the identity cos(a -b) = cos(a) cos(b) + sin(a) sin(b) reveals 1 r r i=1 cos ω ⊤ i (xx ′ ) = ϕ r (x) ⊤ ϕ r (x ′ ), the above bound can be rewritten asP ϕ r (x) ⊤ ϕ r (x ′ ) -k(x, x ′ ) ≥ ϵ ≤ 2 exp - rϵ 2 2 . Also, k r (x, x ′ ) = ϕ r (x) ⊤ ϕ r (x ′ ) is by definition a normalized kernel, implying that ∀x ∈ R d : ϕ r (x) ⊤ ϕ r (x) -k(x, x) = 0.As a result, one can apply the union bound to combine the above inequalities and show for every sample set x 1 , . . . , x n :P max 1≤i,j≤n ϕ r (x i ) ⊤ ϕ r (x j ) -k Gaussian(σ 2 ) (x i , x j ) 2 ≥ ϵ 2 ≤ 2 n 2 exp - rϵ 2 2 .Considering the normalized kernel matrix 1 n K = 1 n k(x i , x j ) 1≤i,j≤n and the proxy normalized kernel matrix 1 n K = 1 n ϕ r (x i ) ⊤ ϕ r (x j ) 1≤i,j≤n , the above inequality implies thatEQUATION)Leveraging the eigenvalue-perturbation bound in [64] , we can translate the above bound to the following for the sorted eigenvalues λ 1 ≥ • • • ≥ λ n of 1 n K and the sorted eigenvalues λ 1 ≥ • • • ≥ λ n of 1 n K n i=1 ( λ i -λ i ) 2 ≤ 1 n K - 1 n K F Mode #Incompatibility with non shift-invariant kernels. Our analysis targets a shift-invariant kernel, which does not apply to a general kernel function, such as polynomial kernels. In practice, many ML algorithms rely on simpler kernels that may not have the shift-invariant property. Due to to specifics of FKEA framework, we cannot directly extend the work to such kernels. We leave the framework's extension to other kernel functions for future studies.Reliance on Embeddings. FKEA clustering and diversity assessment metrics rely on the quality of the underlying embedding space. Depending on the training and pre-training datasets, the semantic clustering properties may change. We leave in-depth study of embedding space behavior for future research.This section details the results of cluster analyses conducted on various real-world datasets, including FFHQ, AFHQ, MSCOCO, and Fashion-MNIST. Each dataset's results are organized into clusters identified by RFF method in FKEA evaluation.In this section, we evaluate the quality of clusters obtained from the ImageNet dataset as the number of samples n varies. Specifically, we compare clustering results for 10k, 50k, 100k, and 250k samples. Figure 7 illustrates the first four modes derived from the FKEA framework.At n = 10k, the clusters exhibit noise and often merge unrelated modes, as seen in Mode 2, where elephants and foxes appear within the same cluster. As n increases, the clustering quality improves, becoming more coherent and meaningful. This trend is particularly evident in Modes 1 and 2, where the clusters more accurately reflect distinct semantic groups.These findings highlight the importance of scaling VENDI and RKE scores, as computational overhead becomes a critical factor in assessing the diversity of generative models. Scaling these metrics allows for a more efficient evaluation, especially when dealing with large datasets and high sample counts.In this section we report the FKEA scores for various generative models on ImageNet dataset. Table 3 evaluates the diversity scores of various ImageNet GAN models using the FKEA method applied to VENDI-1 and RKE, with potential extension to the entire VENDI family. The comparison includes baseline diversity metrics such as Inception Score [12] , FID [7] , Improved Precision/Recall [10] , and Density/Coverage [11] .In addition to running clustering on ImageNet dataset, we also applied FKEA with varying Gaussian Kernel bandwidth parameter σ to other datasets. The results are presented for FFHQ (Figure 8 ), AFHQ (Figure 9 ) Microsoft COCO (Figure 10 ) and Mono/Color versions of F-MNIST [50] (Figures 11 and 12 ) up to top 8 modes.The experimental setup is similar to figure 3 with the only change is optimised bandwidth for each dataset, since datasets differ in number and typicality of the samples.Even though DinoV2 is a primary embedding in our experimental settings, we acknowldge the use of other embedding models such as SwAV [24] and CLIP [23] . The resulting clusters differ from original DinoV2 clusters and require separate bandwidth parameter finetuning. In our experiments, SwAV embedding emphasizes object placement, such as animal in grass or white backgrounds, as seen in Figure 17 . CLIP on the other hand clusters by objects, such as birds/dogs/bugs, as seen in Figure 18 .These results indicate that FKEA powered by other embeddings will slightly change the clustering features; however, it does not hinder the clustering performance of RFF based clustering with FKEA method.To highlight the compatibility of FKEA across diverse embedding spaces, we conducted convergence experiments on various text and image embeddings. Figure 19 presents the convergence results of the VENDI and RKE scores, comparing both FKEA and non-FKEA counterparts. Our findings show that the convergence remains consistent across different embedding spaces, demonstrating the robustness of the proposed method.To understand the applicability and effectiveness of the FKEA method beyond images, we extended our study to text datasets. We observed that clustering text data poses a more challenging task The process of evaluating text clusters is not straightforward and often varies significantly based on human judgment and perception.To visualise the results, we use YAKE [75] algorithm to extract the keywords in each text mode and present the identified unigram and bigram keywords. We demonstrate that the results hold for text datasets and identified clusters are meaningful.Table 4 displays the identified clusters associated with Wikipedia article titles and keywords analyzed using the FKEA method. Identified mode 1 correlates most with historical figures/events/places. Mode 2 clusters smaller villages and rural regions together. Mode 3 is exclusively about people in sports, such as athletes and referees. Mode 4 visualises various music bands and albums. Lastly, mode 5 presents the articles about sports events, such as football leagues.Table 5 outlines the largest modes identified within a news dataset analyzed using the FKEA method, with a detailed focus on the content themes of each mode. The most dominant mode is associated with topics related to crime and police activities, indicating a frequent coverage area in the dataset. Mode 2 is closely correlated with President Obama, reflecting a significant focus on political coverage. Mode 3 pertains to dieting, which suggests a presence of health and lifestyle topics. Mode 4 is linked to environmental disasters, highlighting the dataset's attention to ecological and crisis-related news. Finally, Mode 5 deals with plane crash accidents, underscoring the coverage of major transportation incidents.Table 6 delineates the distribution of genres and production types within a dataset of movie summaries analyzed using the FKEA method. The first mode predominantly covers drama TV shows without focusing on any specific subtopic, indicating a broad categorization within this genre. From mode 2 onwards, the features become more distinct and defined. Mode 2 specifically represents Bollywood movies, with a significant emphasis on the Romance genre. Mode 3 is dedicated to clustering comedy shows. Mode 4 is exclusively associated with cartoons, evidenced by keywords such as "Tom & Jerry". Lastly, mode 5 clusters together detective and crime fiction shows.In this section, we present additional experiments on the Kinetics-400 [58] video dataset. This dataset comprises 400 human action categories, each with a minimum of 400 video clips depicting the action. Similar to the video evaluation metrics, we used the I3D pre-trained model [61] which maps each video to a 1024-vector feature. Figure 20 , the first mode captured broader concepts while the other models focused on specific ones. Also, the plots indicate that increasing the number of classes from 40 to 400 results in an increase in the FKEA metrics. Justification: The submission comes along with demo code to generate clusters and compute the scores.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.This computation complexity is the minimum known achievable cost for multiplying n × n matrices which we prove to lower-bound the complexity of computing matrix-based entropy scores. 


Decision: The authors proposed an algorithm to evaluate the diversity of generated data in generative modelling. Most metrics used in applications rely on reference datasets and reference-free entropy scores VENDI and RKE are too computationally intensive in high dimensional settings. In this paper, the authors use the random Fourier features framework to propose an approximation of these scores reducing significantly the computational complexity. 

The paper was reviewed by three experts who all gave a borderline accept score highlighting in particular that the paper is well written and that the contribution is useful for the community. The main limitation were related to the experiments to highlight the claim of the authors. 

Providing efficient ways to evaluate generative models is a crucial challenge in particular for practitioners as these models are at the heart of a very intense research activity. The authors provide a very comprehensive procedure to evaluate the diversity of generated samples based on  random Fourier features supported by theoretical guarantees and extensive simulations using toy data, image and video data. The additional experiments run during the rebuttal improve the paper and confirm the reduced complexity which makes the proposed approach interesting in large scale settings where  original entropy-based diversity evaluation scores are not available.

Reviews: review: paper_summary: paper_summary: This study proposes a computationally efficient metric for evaluating the performance of recent generative models. It highlights the limitations of using reference data, which can restrict the applicability of evaluation methodologies, and instead suggests a method utilizing kernel functions without references.
Experiments demonstrate that the proposed metric shows similar trends to existing metrics across various modalities and datasets, and qualitatively reflects the characteristics of the data in the metric computation process.
strength_weakness: strength_weakness: * Compared to existing metrics, the speed improvement in performance measurement is not experimentally specified.
* The numerical alignment with existing metrics is not provided.
questions: questions: - Please provide the extent of the speed improvement over existing method in actual experiments.
- Provide experimental results based on the capabilities of different embedding models (e.g., compare performance measured using BERT for text with that using text-embedding-3-large).
- Measure the correlation between the existing metrics and the proposed metric in your experiments.
limitations: limitations: Yes
paper_summary: paper_summary: The work introduces a new method called Fourier-based Kernel Entropy Approximation (FKEA) to evaluate the diversity of data generated by generative models. Traditional evaluation metrics for generative models often rely on reference datasets, which may not always be available or suitable. Recently, reference-free entropy scores like VENDI and RKE have been proposed but suffer from high computational costs, especially with large-scale models.

FKEA addresses this issue by leveraging the random Fourier features framework to reduce computational complexity. It approximates the eigenspectrum of the kernel matrix to estimate entropy scores efficiently. The method utilizes proxy eigenvectors derived from FKEA to identify modes in the diversity assessment of generated samples.
strength_weakness: strength_weakness: 1. Evaluation seems to be limited to simple datasets. How would the method perform on complex image and video datasets? 
2. The reported metrics cover the basics of establishing the advantage of this method. Any other metrics that can be used to establish clear dominance of this method over existing methods?
3. Qualitative results are hard to follow.
questions: questions: see weaknesses section.
limitations: limitations: see weaknesses section.
paper_summary: paper_summary: This paper introduces the Fourier-based Kernel Entropy Approximation (FKEA) metric, which efficiently evaluates the diversity of generated samples. The key contributions of this work are twofold: (1) Compared to existing diversity metrics such as VENDI and RKE, the proposed metrics (i.e, FKEA-VENDI and FKEA-RKE) are more computationally efficient, reducing complexity to $O(n)$ for a sample size $n$. (2) The proposed metric is reference-free and can be used to assess the performance of large-scale image, text, and video datasets for generative models. Extensive experimental results demonstrate the effectiveness of the proposed metric.
strength_weakness: strength_weakness: - The proposed evaluation method relies on a number of parameters, e.g., $\sigma$ and $r$. The values of these parameters seem to vary significantly according to different tasks. For example, the authors adopt $2r=4,000$ and $\sigma=7$ for the experiment on the MNIST dataset, and $2r=16,000$ and $\sigma=25$ for the experiment on the ImageNet dataset. How these parameters influence the evaluation results remains unexplored. (See Questions 1 and 2)
- The paper could be improved by incorporating a toy example to illustrate the difference between reference-dependent and reference-free metrics. The discussions in Sections 1 and 2 are not concrete enough. (See Question 3)
- One of the key contributions of this paper is the theoretical improvements in the computational complexities of the FKEA-based metrics. An evaluation time comparison between FKEA-based metrics (e.g., FKEA-VENDI and FKEA-RKE) and their original metrics (e.g., VENDI and RKE) is missing. (See Question 4)

---

**Minor Point**

- There is a typo (i.e., $x \to \mathbf{x}$) in Line 157.
questions: questions: 1. The proposed metrics require stochastic approximations on expectations (i.e., $\mathbf{x}$ and $\boldsymbol{\omega}$). Could the authors report the variances of the evaluation results? Additionally, can the authors provide guidelines for choosing a sufficient number of samples?
2. Is there any underlying intuition for selecting $\sigma$ and $r$? Will an inappropriate selection of $\sigma$ and $r$ lead to an inaccurate assessment of the true generative quality?
3. Could the authors provide experimental results to demonstrate that reference-dependent metrics (e.g., Recall and Coverage) may fail to measure sample quality, while reference-free metrics may be successful?
4. Could the authors provide an evaluation time comparison between the calculation of FKEA-based metrics and their original metrics? For example, an evaluation time comparison between FKEA-VENDI and VENDI would be helpful.
limitations: limitations: The authors have allocated a section (i.e., Line 292) for the discussion of potential limitations.

Comments: title: Thank you for your feedback
comment: Thank you for addressing my concerns. I'd stand by my initial rating.
Dear Reviewer ymdX,

We sincerely thank you for your feedback on our rebuttal. We are pleased to hear that our responses and the additional experimental results were satisfactory. As mentioned in the rebuttal, we will include these new results in the revised draft.
If any further questions or comments arise during the remaining two days of the discussion period, we would be more than happy to address them.

Thank you once again for your thorough review and thoughtful consideration.
Dear Reviewer ohKy,

We sincerely appreciate the time and effort you have invested in providing feedback on our work. As we approach the end of the discussion period, with only two days remaining, we wanted to ensure that all of your questions and concerns have been addressed. If there are any aspects of our submission or rebuttal that still require clarification, please let us know. We would be happy to provide any additional information or explanations.
Dear Reviewer NG2d,

We sincerely appreciate the time and effort you have invested in providing feedback on our work. As we approach the end of the discussion period, with only two days remaining, we wanted to ensure that all of your questions and concerns have been addressed. If there are any aspects of our submission or rebuttal that still require clarification, please let us know. We would be happy to provide any additional information or explanations.
Thank you for the response. The additional results are a nice enhancement to the paper. I have no further questions and will keep my score.