Statistical Inference for Generative Model Comparison
Authors: Zijun Gao
Han Su
Yan Sun
Venue: Accepted by TMLR ()

Abstract: Generative models have achieved remarkable success across a range of applications, yet their evaluation still lacks principled uncertainty quantification. In this paper, we develop a method for comparing how close different generative models are to the underlying distribution of test samples. Particularly, our approach employs the Kullback-Leibler (KL) divergence to measure the distance between a generative model and the unknown test distribution, as KL requires no tuning parameters such as the kernels used by RKHS-based distances. And the relative KL divergence is the only $f$-divergence that admits a crucial cancellation of the hard-to-estimate term to enable the faithful uncertainty quantification. Furthermore, we extend our method to comparing conditional generative models and leverage Edgeworth expansions to address limited-data settings. On simulated datasets with known ground truth, we show that our approach realizes effective coverage rates, and has higher power compared to kernel-based methods. When applied to generative models on image and text datasets, our procedure yields conclusions consistent with benchmark metrics but with statistical confidence. The source code to reproduce our experiments is available at https://github.com/sylydya/compare-generative-models.

Published in Transactions on Machine Learning Research (March/20P)
Statistical Inference for Generative Model Comparison
Zijun Gao" Marshall School of Business, University of Southern California
uijunggco@marrshallss....e
Han Su*
suhan@@ail bru.. edu.ccn
School of Statistics, Beijing Normal University
Yan Sun"
yun.sun @njji. edu
Department of Mathematical Science, New Jersey Institute of Technology
Reviewed on OpenReview: https: // openreview. net/ forun? id= PXL65Bah0q
Abstract
Generative models have achieved remarkable success across a range of applications, yet their evaluation still lacks principled uncertainty quantification. In this paper, we develop a method for comparing how close different generative models are the underlying dis- tribution of test samples. Particularly, our approach employs the Kullback-Leibler (KL) divergence to measure the distance between generative model and the unknown test dis- tribution, KL requires tuning parameters such the kernels used by RKKHS-based distances. And the relative KL divergence is the only f f-divergence that admits a crucial cancellation ion of the hard-looestimmaee term to enable the faithful uncertainly quantification Furthermore, we extend our method to comparing conditional generative models and lever- age Edgeworth expansions to address limited-data scttings. On simulated datasets with known ground truth, we show that our approach realizes effective coverage rates, and has higher power compared to kernel-based methods. When applied generative models im- age and text datasets, our procedure yields conclusions consistent with benchmark metries but with statistical confidence. The source code to reproduce our experiments is available at
Keywords: Uncertainty quantification, Generative model evaluation, Edgeworth expansion, f-divergence
1 Introduction
Generative models have achieved remarkable success across wide range of applications, demonstrating their versatility in areas such image synthesis, natural language processing, and scientific discovery (Achiam ct al. 2023 Goodfellow et al.) 2014 Van Den Oord et al.] 2016a Karras et al., 2020). As the number of generative models continues to grow rapidly, practitioners increasingly face the challenge of selecting the most appropriate model. This underscores the noed for principled methods of performance evaluation and model comparison.
Despite the advances in model development, approaches for assessing and comparing generative models remain relatively underexplored. One class of evaluation methods rely human judgment, which are inherently subjective and difficult to scale. An alternative line of methods use standardized benchmark tests [Liu et al. 2024 Gallifant et al.] 2024) (such as ATME 2024 and MATH 500), which are limited to output types for which such benchmark tests exist. In addition, various quantitative metrics have been proposed for different types generated outputs, such as Wasserstein distanco-based metric (c.g., Fréchet Inception Distance, FID) for images and perplexity for texts. However, these metries lack uncertainty quantification,
* "Alphabetical order

---

Published in Transactions on Machine Learning Research Mlarch/2020
FID = 4.67
FID 4.16
Relative Score:
8(IP1, P2) = - (L I1pp) :) + KL(PI|P2)
Unbiased Estimator: 3(F,,P2) -17.02
Confidence Interval: (a) = (-17.40,-16..)
P(8(PL, P2) E (a)) - a, a == 0.1
P1
P2
Figure 1: An example of our method applied to comparing the diffusion models (DDIMs with different numbers of denoising steps S). Here, P represents the distribution of the test images, P1 corresponds to the DDIM model with $ = 50 denoising steps, and P2 corresponds
DDIM model with 100 denoising steps. Our method demonstrates that confidence interval P2)) ) significantly negative, indicating that P2 with S 100 achieves significantly better performance. While consistent with the FID reported in [Song et al. (2021, our method further quantifies the statistical significance of the performance difference, which FID cannot. provide.
the relative score (f)),
making difficult to determine whether observed differences reflect true performance gaps or arise solely from variability of the test sample
The lack of satisfactory model assessment methods can be partially attributed to the challenge that evaluating generative models requires comparing output distributions rather than individual data points. For predictors or classifiers, the performance can be directly measured by comparing the generated outputs to the associated true labels. In contrast, the quality of a generative model is determined by how closely the distribution of generated data matches that of the input data, rather than the similarity between gencrated data points and input data points (known the reconstruction error). Moreover, generative models often produce high-dimensional outputs? making the distríbution comparison more challenging.
In this paper, we develop methods for comparing generative models with uncertainty quantification. Specif- ically, we make the following contributions:
We propose method for evaluating and comparing generative models based on the relative score of KL divergence. Focusing on the relative score enables the cancellation of hard-to-estimate quantity in the absolute score, leading to favorable statistical properties.
We develop unbiased estimator in the form of first-order U-statistic that achieves convergence at the parametric rate. We explicitly characterize the asymptotic distribution of our estimator. To improve the finite sample performance of our method, we refine the limiting distribution analysis via Edgeworth expansion, which leads to faithful coverage even with very small sample size.
We further extend our framework from unconditional to conditional generative models, which are central in practice (e.g,, text-to-image models, language models). This allows principled statistical comparison of conditional distributions.
On simulated datasets with known ground truth, we demonstrate that our approach constructs faith- ful confidence intervals and achieves higher power compared to baselines. Furthermore, we demon- strate the offectiveness of our method in evaluating generative modcls on the CIFAR-10 dataset and assessing large language models (LLM) on Wikitext-2 and TriviaQA data sets.
'In practice, metrics such as FID are usually computed given a dataset for evaluation. In our work, we view the ground truth distribution as an unknown population distribution, and the dataset for evaluation is viewed as a finite sample from the distribution. This finite-sample approcimation is the main source of the uncertainty that we quantify in our work. From this perspective, FID is viewed as the Wasserstein / Fréchet distance between the transformed distribution induced by the Inception model.
2The output is typically altre-iigh-diiieessoonall for example, 1080p resolution image consists of around 2 10 pixels with (upproximately) contimuuus value. Particularly, image is represented as a matrix of size d2, where d d1 du is the total number of pixcls. Each pixcl cousists of three color channels (e.g., RGB), where each channel takes integer values ranging from 255. result, single pixel can represent 256 16.7 x 109 possible color combinations. The pixel values can effectively be regarded as continuous.
2

---

Published in Transactions on Machine Learning Research Mlarch/2020)
Organization. In Section 2 formulate the problem of evaluating generative models. In Section 3 we introduce the concept of relative score which enables a. key cancellation, and propose our uncertainty quantification method. In Section 4 we extend the method to conditional generative model comparison and the cases with limited test data. In Section 5 we evaluate the numerical performance of our methods using both simulated and real image and lext datasets. All prools and additional literature, method details, and empirical results are provided in the Appendix.
Notations. Let denote the space of test data, and let P represent the true target distribution. Let IPr denote the distribution of data generated by a generative model. If the generative process of the generative model involves sampling random noise vector, we use di to denote its dimension. Similarly we define IP and d2. Let P. represent the empirical distribution of n observations from P, and similarly for Pi,n, P2... We denote densities by lower-case letters, c.g,, as the density of P. We use da to denote the density of the standard multivariate Gaussian density in IR.. For I E Rd, we use ||>]|| to denote its l norm.
2 Formulation and background
2.1 Comparison of generative models
In this paper, we focus on the case where a set of neest test data points Y,, 1 S S Phest independently and identically distributed (i.i.d.) from the target distribution P, are provided. We aim to compare generative models using the test dataset. For conciseness, we focus on the comparison of two models, denoted by IP1 and I22, while the generalization multiple-model comparisons straightforward.
Generative models are designed to produce samples whose distribution approximates the true underlying distribution P. To evaluate generative model P1 quantitatively, a dissimilarity metric between IP and Pi is computed. In this work, we use the KT. divergence, also known as relative entropy, for the following reasons: (1) KL divergence the only f-divergence that satisfies a cancellation property (see Proposition D. which enables unbiased estimation and valid uncertainty quantification; (2) KL divergence strongly penalizes mismatches in regions where one distribution assigns probability mass and the other does not, desirable when but consequential events shall drive conclusions; (3) unlike kernel or representation-leannii basedd metrics, KL divergence requires neither choosing a kernel nor learning latent features, reducing the risk of uninformative comparisons caused by suboptimal hyperparameter choice or insufficient luning.
Assumption 2.1. Suppose the lest dala distribulion P and the generaled data distribution P, admil densilies, and I is absolutely continuous with respect to Pl.
Under Assumption 2.1 we formally define the absolute score for the generative mode 3 associated with P as the negative KL divergence between IP and P,,
p(y) ply) dy,
s(Pi) = -KL(P)))) =
(1)
where we denote the density of P. P by p, ir respectively. larger absolute score s(fr)) indicates better performance of the generative model P,.
In this paper, we focus on comparing generative models whose sampling density evaluated a test data point is accessible. Many existing models satisfy this property. Details about the computation of the density function for different generalive models are provided in Section
Generative models for image. A significant proportion of generative models for images, including variational auto-encoders (Kingma 2013), autoregressive models (Van Den Oord et al.] 2016b)
normalizing flows (Rezende & Mohamed, 2015 Dinh et al. 2016), diffusion models (Ho et al.) 2020
Song & Ermon 2019 Song et al.) 2021 consist of a forward and a reverse process. Any image generative model that admits an accessible and invertible reverse process, such as normalizing flows
Forr conciseness, the generating distribution represent corresponding generative model. For example, may refer the generative model Pt.
3

---

Published in Transactions on Machine Learning Research (March/200 (
Denoising Diffusion Implicit Model (DDIM) (Song et al. 2021 permits the density evaluation at test data point.
Chen al. 2018
Generative models for text. Autoregressive language models, which generate text token by token conditioned on previous context, are the most widely used L.LSs nowadays. Let V denote the TL), ri V denote response sequence, and let ri:i denote the first
vocabulary. Let (r1, to the i-th tokens of r. Let P denote the ground truth probability of responses. An autoregressive language model defines the probability of the next token given previous tokens as PP (1(4t11/r:s). By the chain rule of probability, the joint probability of the sequence is
L-1
Pi(r) = II Pi(raalra)) E=0
where Pa(ry (ra). For open-source LLMs, the estimated probability Pi(r) at test data point is typically accessible through the forward pass.
2.2 Related works
We review literature on generative model evaluation based on distributional closeness. We acknowledge that generative models are also evaluated along other dimensions (e.g., overfitting, diversity, fidelity, memoriza- tion), which are important in practice but lie beyond the scope of this work.
Vasserstein--distaace--ase evaluation Wasserstein-p distance (Villani et al. 2009 is defined as
(F, inf rEI(P.)))
BLP2 B( ||| vl
where [(P, is the set of couplings between IP and PI. There are several challenges of performing inference for evaluation methods based the Wasserstein-p distance. First, Del Barrio & Loubes (2019) shows tha
Vn W3 (P... E(W3 (Pr. S (0,00
where the asymptotic variance (TP, P.)) can be estimated consistently using a plug-in estimator. The issue is that the center (P,, P1.,.] is different from the desired (FP PP), and the gap between EFW PIn)) and (P, IP1) scales as n 1/d the joint asymptotic distribution of (W) n. (P,... IP,.)) )) is required. It remains unclear whether (P,, PI... and (T(P,,, n. P2.n) are osiiplll jointly Gaussian, as well what the exact form of their covariance matrix - Third, subsampling methods Dümbgen (1993) can be employed for conducting inference for Wasserstein-p distances; however, subsampling is computationally expensive, especially given that the Wasserstein-p distance already difficult to compute. For a comprehensive review of these issues, see Panaretos & Zemel (2019).
-1// Villani et al. (2009). Second, for the relative Wasserstein-2 distance W:3 (I,, P1),
FID WID/IS-based evaluation We review the inception score (IS) Salimans et al. 2016 and the Frechet Inception Distance (FID) (Heusel et al. 2017; Jayasumana et al. 2024 Jiralerspong et al. 2023 two most commonly used quantitative scores for generative models. IS evaluates the quality of a generative model by applying a separate, pretrained image classification model to batch of images generated by the model. IS is maximized when the classifier confidently predicts a single label for each image, or when the predictions are evenly distributed across all possible labels. The quality of IS depends heavily on the quality of the classifier (if the classifier consistently outputs a single label for all images, the IS becomes uninformative). Another disadvantage that IS does not compare generated images to test images. FID compares the distribution between the distribution of test images and that of generated images. Mathematically, FID aims to approximate the Wasserstein-2 distance between the two distribution in two steps: (1) mapping the
*CLT results of general Wasserstein-p distance are largely unknown. 5he off-diagonal values the covariance matrix non-zero because both (P P1.n) and WP P2.n) depend on Pp-
4

---

Published in Transactions on Machine Learning Research (March/20P))
real and generated images to R separately by passing them through the final layer of an image classifier to extract essential features; (2) fitting multivariate Gaussian distributions to the transformed data in Rd and computing the Wasserstein-2 distance between these multivariate Gaussians. The approximation accuracy depends on how well the transformation to R captures the data characteristics and the quality of the multivariate Gaussians fit. to the transformed data (the covariance matrix lypically not diagonal and the estimation involves (dP) elements). Chong & Forsyth (020)) shows that FID could be significantly biasod in finite sample.
Kernel-based evaluation There is line of kernel-based evaluation metrics which admit uncertainty quantification (Binkowski et al. 2018 Liu et al.) 2016 Chwialkowski et al. 2016 Bounliphone et al. 2016 Kanagawa et al.) 2023). However, nernel-based evaluation hinges on having a powerful kernel, which is challenging for complex, high-dimensional data such as images and text. In particular, as we show in the simulations, even carefully designed kernels may fail to capture all fine- sine-grained distinctions. Moreover, kernel-based methods are computationally expensive, requiring quadratic pairwise computations, whereas our method scales linearly in sample size. For kernelized Stein discrepancies (Liu al. 2016 Kanagawa ct al. 2023 scores of generative models are roquired, which can be expensive to compute. For large language models (LLMs), evaluating the score entails back-propagation through the network, which is substantially more costly than the single forward pass required by our method.
Likelihood-ratio-bbseedd evaluation There is thread of works in using likelihood ratio for comparing statistical models (Neyman & Pearson 1933 Wilks 1938; Vuong 1989). Given two candidate families of distributions (two statistical models), first fitting cach model to the data and then evaluating the difference in the fitted models' log-likelihoods. However, for modern generative models (Song et al., 2021 Dubey et al., 2024), this fitting step is typically computationally expensive or even prohibitive. In contrast, our approach operates on pre-trained generative models and requires no additional fitting. In addition, we complement. this line of work by offering con- ditional comparisons (Section 4 which are important in scenarios where poor performance within specific subpopulations is especially consequential dangerous. Finally, while this literature adopts log-likelihood to maximize test power, work with the KL divergence mostly because it is the only f-divergence that admits the cancellation property (Proposition
ratioood--atio-bnne
tests determines the more suitable family by
1. crucial for valid inference.
3 Relative score for generative model comparison
Despite the popularity of the KL-divergence (our absolute score), its estimation and inference are considerably challenging. According to Zhao & Lail (2020), the minimax optimal rate for estimating the KL divergence between two densitics in Rd, based on a sample of size Neest from each density, scales as slow as neest addition, the asymplotic distribution of the KL-divergence is typically intractable except for special cases bootstrap or subsampling methods are called for to
2/d In
(Belov & Armstrong 2011), and computationalll--eett conduct inference based on the KI.-divergence (Arizono Ohta 1989..
Instead of investigating the absolute scores of two generative models separately, we propose to directly study the relative score, the difference between their absolute scores, defined as
= - (L(PP (LII)()
(2)
The relative score aims to quantify the performance gap between the two generative models. If (f 0. it implies that (L(PIIP,) < KL(P[[(22) and we conclude that P, is superior to IP2.
In contrast to the absolute score, the relative score benefits from a nice cancellation of some hard-to-estimaal term, facilitating its estimation and inference. Explicitly, the absolute score [ contains two terms, with f log(log(p())) dI)(y) being less tractable than f (y)) dIP()), as the generating mechanism Pi (γ) is essentially known, i.e.,
s(P,) = - -(f log (p(v)) dF(r)-
flog log (pi(v)) dP(w))
Challenging
Tractable
5

---

Published in Transactions on Machine Learning Research Mlarch/20200
By the definition
2 ((P1,P2) equals
- -( (log(o(w)) dP log(Ei (v)) dP(v))
(/
log(p(()) dP(y)
log( d((y))
Challenging
Tractable
Challenging
Tractable
(3)
(og( dP()) log(f2(U)) dP(y).
Tractable
Tractable 2
Here the challenging term log(p(v)) dl(y)) appearing in both KL(P][[1) and (L(PI(P2), cancels out. The remaining term f log(P1 (og)_) (y)) dlP(y) is the expectation of an effectively known function log(21 (v)) (og( regarding the test data distribution. Therefore, the relative score can be efficiently estimated using a first-order U-statistic based on a set of test data points, detailed in Section 3.1 below.
We conclude this section by showing that the attractive cancellation (3) in the relative score is unique our choice of KL divergence. For convex function f [0, +00) - (- 100, +ool such that f(x) is finite for all > 0, f(1) O, and f(0) lim,-o+ [((), the I-divergence of P from IP is defined Dy(PI(P)) ==
S
P1 (v)dy. KL-divergence is special case of f-divergence with f(x) x x pi(y)
p(v)
Proposition 1. For an f-diveryence with Ct if there exists function g such that for any Pl, Pe, P,
Dy(PI(P:) - (P(()()2) = 9PP,,
(4)
then there erists B 2 0 such that f(x) = Bx log(z)), i.e. Dy(11((1) = (KL(P)))))
We prove Proposition by (1) reducing it to the Cauchy functional equation problem through multiple
rounds of re-parametrization: (2) applying the uniqueness of the solution to the Cauchy functional equation.
Details are provided in Section B of the Appendix.
3.1
Estimation and Inference of Relative Score
3.1.1
Estimator
By 3 for generative models with accessible (Y() and (2(Y)), we estimate the relative score by the first- order U-statistic (sample mean) the test dataset,
Thest
8 8(P,, PP) = Thaest
(og()(())) log(@> (Y:)).
(5)
According to the standard property of U-statistics, we establish the following unbiasedness result.
Proposition 2. The eslimator 6(1P,,, P2)) in (5) is unbiased, i.e., E [(P Pa)] 6(1P, P2).
In Section of the Appendix, we detail the computation of our estimator for both image and text generative models discussed in Section 2 along with several acceleration techniques to improve the computational efficiency.
3.1.2 Inference
We describe the asymptotic distribution of the estimator in 5 in the following theorem.
Theorem 3.1. Let V Var (log(š) (Yi) - (og((((Y))))) V <0,
Niestt (8(P), P2) - SCPP, P2)) $ $0,V..

---

Published in Transactions on Machine Learning Research (March/200)
The detailed proof is provided in Section B of the Supplementary Material. Other methods that provide uncertainty quantification (Bounliphone et al. 2016 Liu et al. 2016 Kanagawa et al. 2023 focus non-KT, distances (e.g., kernel-based distances) and establish the joint asymptotic distribution of estimated absolute scores (S(P1). ((P2)). In contrast, for the KL divergence in our method, the absolute score cstimators 8((P;) and a(P2) are heavily biased with intractable asymplotic distribution, but their difference enjoys the cancellation and still converges at the -1/2 rate to normal limit.
By the law of large numbers, the empirical variance of log(ß1 (Y.)) 1o---logo -log(p2 (Y))), denoted by V, is consistent estimator of V. Using Slutsky's theorem (see e.g.: Lemma 2.8 of Van der Vaart (2000)), ) derive the following corollary of Theorem 3.1
Corollary 3.1. IfO < V < xo, then
8(P,,, - 8((P1,P2) (5
(6)
V naest
Corollary 3.1 allows us to perform statistical inference on the relative score, enabling the determination of the better generative model with a specified level of confidence. Explicitly, let a E (0, 1) be the confidence level, and we consider the following confidence interval (C1) of the relative score, denoted by CI(a),
(P) - -1--2
PP) + 91-
V
(7)
Pocet
where 41-# denotes the upper 1 - a/2 quantile of standard normal. The following statement states the validity of the confidence interval.
Corollary 3.2. Under the conditions in Corollary 3.1 for any (0, PP (D(P),,P)) Ci(a)) 1
4 Extensions
4.1 Conditional generative model comparison
Our method can also be exlended to the comparison conditional generative models. Suppose that the data (X,, YJ)= are generated from joint distribution IP with density p(X, Y). Conditional generative models aim to approximate the conditional distribution p(Y]X) via p(Y]X). Then p(Y]X)p(XX serves as approxi- mation of the joint distribution Given two conditional generative models Pi,P2 that aim to approximate the conditional distribution p(y(=)) by (vlæ) and Åa (g(lx, similar to the Section 3 we consider the absolute
score (L((p x)ll (x))) Then the corresponding relative score 6(1P, P2) can be defined as
KI. (p(2, I) (vla)p(z)) + KL (p(x, v) (1l
f (log p (vl=) log Pa (v(r)) p(z, dddde
where the hard-to-est entropy term also cancels out. Thus, we can define the unbiased estimator by:
S P2)
Σ log pa (ygl log (2
which is again a first-order U-statistic. All thcoretical results in Section 3 naturally extend to the conditional setting.
6I the setting conditional generative models, conditioned quantity X is usually given, therefore we don't need approximate p(X).

---

Published in Transactions on Machine Learning Research Mlarch/2020)
4.2 Edgeworth expansions for limited test data
Corollary 3.2 ensures asymptotic correct coverage rate of the CT(a), but when we have limited test data (n S 100) (which may arise in settings with costly human labels, slow data collection, online experiments with early stopping), or non-Gaussian output (i.e., PP deviates from the normal distribution), then CT(a) may not always provide reliable coverage rate. In this subsection, we focus on further theoretical discussions on the inference of the relative score. By introducing a high-order expansion for the distribution of testing statistics, called Edgeworth Expansions (EEs), we can obtain more accurate statistical inference results. We first present the superiority of EES over classical Central Limit Theorem (CLT, c.g., Theorem 3.1 with respect to constructing CI of the relative score 1 P2) when sample size n is small, as shown in Figure The detailed setting is given in Section 5.1.1
Normal: Coverage Rate of 90% CI
Normal: CI Length
1.00
Coverage Rate
0.9
CI Length
EEs
0.50
CLT
EEs
0.25
Level
0.90
0.7
0.00
10
10a
10
10 10'
n
Figure 2: Coverage rates and length of confidence intervals obtained by Edgeworth Expansions Theorem across different sample sizes n using simulated data generated by
12 and Central Limit
13 with 0.07.
By utilizing information from higher-order moments, such skewness and kurtosis, EEs can yicld more precise limiting distributions and faster convergence rale against CLT. Specifically, we revisil the proof of Theorem 3.1 of Z-statistics Zn (V/neea)) - 8(P,, S(P,, P2)] in Theorem 3.1 For simplicity, we slightly abuse the notations, lct {X; = ( E(X) μ, 0 < Var(X)= o² < oo, and let X, = n Σ X₁. By the independence of X,, the characteristic function (c.f.) of Zn has the expansion form based on the first and second moments of X₃:
1o (Y₂)) ~ Fl be i.i.d. relative scores, with
t
t
22, (t) 1x X__
+
(9)
2n
Combined 9 with the Lévy continuity theorem (sce c.g., Theorem 2.13 of Van der Vaart (2000)) we can derive the CLT
Zn Vn((x, n 1) (0,1).
(10)
However, in gencrative models, both (Y;,, (Y.))) and {X,), are usually far away from Gaussian. CLT unify the Gaussian distribution as the asymptotic distribution for the inference of relative scores 6, which may result in unreliable comparison results, especially when is relatively small.
In fact, CLT 19 only performs a second-order moment expansion of the c.f. of Zn, and does not utilize higher-order moments such as skewness and kurtosis. Therefore, one can further expand the c.f. of Z, based on higher-order population moments of X (if exist), such that the limiting distribution of Z,, can be characterized more precisely, further making reliable inferences in generative model comparisons. Based on the calculation of cumulants and the Fourier inversion of viz (t), the following theorem gives the EEs of Zn..
Theorem 4.1. Under certain regularity assumptions on the distribution of X, (details in Section B of the Appendix), the distribution function F,, Fx(x)) of Zn satisfies
K2 K4 Ha(a))())) + Hs(z)d(a)) to(n
F.(x:) = =b(x) -1/2 K3 Hz(x)((x)
24
72
(11)
G(#)
8

---

Published in Transactions on Machine Learning Research Mlarch/20200
where Ki = === =ma/o and K4 ma/o--3 is the skewness and kurtosis of Xs, respectively, Hx(z) is the k-order Hermile polynomial, ((x) is the densily of ((x). The distribution G(z) is called the EFs of Zu. Furthermore,
sup -(()) (n =ER
The detailed proofs of Theorem can be found in Section of the Appendix, Intuitively, the Edgeworth expansions [1] have similar form the Taylor expansion, but are in a stochastic version. The distribution G(x:) from E.Fs is terms of (12)) and O the basis of normal distribution (xx). For comparison, the remainder term for CLT in Theorem 3.1 is o(1). The coefficients of the n n-1/2 and n- term contain higher-order skewness and kurtosis information of relative scores, which more accurately characterizes the approximation error from (2) to (z).
precise and absolutely consistent expansion of Fa (2),, including additional reminder
Theorem enables a more accurate determination of the better generative model compared to the CLT version Similarly, we can obtain the following (1 - a---evel confidence interval Clgrs(a):
V P2) a1 / n
P() - a2\ V
(12)
where 01, 02 are two quantiles of G(z) resulting in the shortest CI length of Clgas satisfying Iat dG(xr) a, s.t. dG(a1) dG((az).
In practice, the variance of relative scores log( (Y))) log(p2(X))) is usually unnnww, one can consider the T-statistics T, = (V (V/np..) 1 Pf, 8(P1. P2)], similar to Corollary 3.1 The details of EEs of T, and the counterpart bias-corrected CT are given in Section B of the Appendix
5
Numerical analysis
5.1
Unconditional comparison
5.1.1
Simulated data
We use P. P.IPr,P. P. to denote the distribution of Y, Y1, and Y2 and generate the data by:
X,,1,X, NNO, la),
Y ~~ AX +B, Y1 ~ AXi + B,
(13)
Y/2 ~ (A + X2 B
where d = 10, A E Raxd is constant diagonal matrix with diagonal elements generated from U(0.8, 1.2), B E Rd is a constant vector generated from (0, Ia), and E R a constant controls the difference wetween P. and P2. We consider e (0.01, 0.02, ..2.....))) generate n = 1000 sample from Y, and construct the confidence interval via 7 with a 0. 1. In this setting, we deliberately design Y, Y1, and Y2 to be multivariate normal, since this is the only multivariate setting in which the Wasserstein-2 distance admits a closed-form expression, which is used by the coverage rate and power calculation below.
We compute the coverage rate and power of our confidence interval over 1000 repeated experiments. The results are shown in Figure 3 Our confidence intervals achieve coverage rates close to the target level. For comparison, we estimate (PP( and (PI(Ip) separately using the k nearest neighbor (kNN) based estimator in Zhao & Lai (200)), take their difference, and construct confidence intervals for (LL(P))) (L(P))(p) using resampling methods including Subsampling (Politis & Romano, 1994] and
Adaptive HulC (Kuchibhotla et al. 2024). We also examine the estimation and resampling- based inference
"When the sample size is relatively small, the EEs provide more accurate statistical inference results, such lower coverage for confidence intervals, lower type-I error for hypothesis testing, and We leave these theoretical problems further research.
8 discussed in Section 4.2 the sample 1000 relatively large, therefore consider the CT using standard CL.T.

---

Published in Transactions on Machine Learning Research Mlarch/20200
0.9-
0.8
KL + Subsampling
0.8
0.7
+ Adaptive HulC
0.7-
Coverage Rate
W2- Subsampling
0.6-
W2 Adaptive HulC
0.6-
MMD
0.5-
target level
Power
0.5
KSD
MMD
Durs
(Auto Encoder)
KSD
Ours (Auto Encoder)
0.2
0.2-
0.1
0.1
0.0
0.00
0.20
0.35
0.0
0.10
0.20
0.30
0.25
KL divergence difference
KL divergence difference
Figure 3: Coverage rate and power of confidence intervals constructed by our methods 7 and cxisting KL divergence and W/2 distance cstimator pairod with resampling methods (Subsampling and and Adaptive HulC), We provide two implementations of our procedure: "Ours" can access Pi. P2, while "Ours (auto-encoder)" uses an auto-encoder to approximate fi. Pa (further details are provided in Section D of the Supplementary Material).
of the Wasserstein-2 distance difference, - -I) (2),, where each Wasserstein-2 distance is es- timated using the empirical distributions. As illustrated in Figure 3 these methods fail to provide faithful confidence intervals. More discussion on why existing estimators fail to provide valid confidence intervals
can be found in Section D.2
In addition, we compare our procedure with other kernel-based methods, where the asymptotic distributions of the estimators are well established. Specifically, we consider the kernel Stein discrepancy (KSD) (Kana- gawa et al. 2023) and maximum mean discrepancy (MMD) (Bounliphone et al., 2016). For KSD, we use the estimator Kanagawa et al. (2023) and use the jackkuile method to estimate the variance. For MMD, we use the estimator and variance estimator in [ounliphone et al. (2016). We compute the coverage rate and power of the confidence intervals over 1000 repeated experiments. As shown in Figure 3 the kernel-based methods provide valid confidence intervals, but their powers are lower compared to our methods.
In Section D of the Supplementary Materials, we illustrate the inference performance of our proposed CLT and EEs methods on more complicated data generation processcs. Across various input distributions of X and various smooth output models of Y, the two methods are similar and achieve desirable coverage for moderately large test sample. For small test datasets, EFs correct the coverage rate to a without sacrificing too much Cl length and power, realizing reliability and detection ability trade-off of generative model comparison when the number of data is small. More discussion when to use CLT intervals versus EFs intervals are provided in Section D
5.1.2 Generative models for image
We apply our methods to real image datasets: evaluating generalive models on the CIFAR-10 dataset. We consider denoising diffusion implicit model (DDIM) (Song et al. 2021), normalizing flow (NF) model Dinh al. (2016). and variational auto encoder (VAE) model (Kingma 2013). For DDIM, we consider the deterministic forward pass (see e.g. Section 4.3 of [Song et al. (2021) as the inverse transformation of the generative model. We compare pretrained models from Song et al. (2021) with different numbers of denoising steps, S = 20, 50, 100, and denote the corresponding generative distribution by PDDDIMS Following Song et al. 2021),, we select the sub-sequence time steps using the quadratic schedule. For the NF and VAE models, we trained them using the default settings provided in the respective repositories"
Our results are consistent with those of (ong et al. (2021) based on FID. Specifically, Song et al. 20021) shows that DDIM models with a larger number of denoising steps S achieve better FID scores, our method also indicates that larger S results in lower KT divergence. Moreover, while the FID scores of IP3o and IP are similar, our results indicate that DDIM model with S = 100 is significantly better.
https://githubt con/ http://riccuteerraannnoon
http://twww..cc//aiiii////ttrrra VAB
10

---

Published in Transactions on Machine Learning Research Mlarch/20200
Comparison of VAE model, NF model and DDIM model with different number of denoising steps S. For DDIM models, a larger number of denoising steps leads to a model with better FID scores and KL divergence. Our confidence interval doesn't cover 0, indicating P100 is significantly better than both $ = 20 and $ = 50.
Table 1:
Model
FID
čI of 6(P Mr PDDIM10)
ČI of relative KID
VAE
175.68
(-8556. 13, -8422.06)
(-1.6107e-01 -154412-001
NF
83.26
(-147888.77 77, -118430.73)
(-7.8398e-02, -7.2044 -7.2446e-02)
DDMM20
6.84
(-39.91, -38.70)
(-2.5521e-03, -1.4817e-03)
DDIMau
4.67
(-17.40, -16.63)
(-2.3001c004, L.8225e04)
DDIM100
4.16
Table 2: Comparison of GPT-2 model variants on the WikiText-2 dataset. Perplexity values are taken from the public leaderboard. Our results show that larger models achieve better performance in terms of KL divergence, consistent with the standard perplexity metric. The confidence interval for 6(PGPT (FP16), PGPT2) includes zero, indicating significant difference between GPT-2 and its quantized version. This supports the practical use of the quantized model.
Model
Perplexity
Čl of 6(P PGPT2)
GPT2-Smas
29.41
(67.763, 71.609)
GPT2-Mcdium
22.76
(28.874, 30.652)
GPrre-Large
19.93
(12.413, 13.351)
GPT2 (FP16)
(-0.002, 0.008)
GPT2
18.34
For comparison, we also evaluate the relative Kernel Inception Distance (Binkowski al. 2018). which evaluates the maximum mean discrepancy (MMD) with third order polynomial kernel, computed InceptionV3 (Szegedy et al. 2015) features of the images. We follow Bounliphone et al. (0016) to compute the relative KID scores and construct the confidence intervals. In the comparison between DDIM30 and 100 the confidence interval for KID covers 0, which fails to distinguish these two models. These results are consistent with our obscrvations in Section 5.1.1 that our method has better power compared to kernel-based method.
5.1.3 Generative models for text
We compare variants of the pre-trained GPT-2 model on the Wikitext-2 data. Particularly, we compare GPT-2 model with different sizes and quantization (models with 12, 24, 36, 48 layers are denoted as GPT2- small, GPT2-modium, GPT2-large and GPT2 respectively, and models with FP16 quantization are denoted by GPT2 (FP16)). We the test set of the Wikitext-2 data, treating each non-title, non-emply line data point, estimate the relative KL divergence by (5) and construct the confidence intervals via The results are summarized in Table 2
Our results yield model ranking that is consistent with the ordering based on the perplexity. But in contrast to perplexity, our framework further provides uncertainty quantification.
Notably, the performances of GPT2 (FP16) and GPT2 are statistically indistinguishable under our metric, indicating that the observed gap is not significant. Our result is consistent. with the common practice of using quantization to improve efficiency without sacrificing performance.
5.2 Conditional comparison
We compare three LLMs, OPT-133 (Zhng al.) 2022 Mistral-7B (Jiang et al. 2023). and Llama3-8B (Dubey et al., 2024) on the validation data of the TriviaQA data set. We treat each language model as conditional generative model that models the conditional distribution p(Auswer/Quest Question). The detailed settings are given in Section D of the Appendix. The results are summarized in Table 3 Our results are consistent with commonly used metrics such as the Fl score.
We further investigate how sample size impacts the confidence intervals of our method by constructing confidence intervals using subset of the validation data. Figure shows the comparison between Mistral-
11

---

Published in Transactions on Machine Learning Research (March/202)
Table Comparison of language models on the TriviaQA dataset.
Model
F1 score
CI of 8(PM. P1amma3s8)
OPT-1.3B
7.47
(-7.19, -7.01)
Mistral-7B
12.24
-0.22,-0.144
Llama3-8B
36.34
7B and Llama3-8B. Even with relatively small sample size (e.g., n 100), method (both CLT and EEs-based) can identify the better model with statistical significance. Since the comparison starts with a very small test sample, our EEs procedure is more appropriate in the starting stage (e.g., for n = 50, the CLT-based interval includes zero, whereas the Baseb interval does not).
Contidence Interval
10
800
1039
2000
Stot
I0000
12313
Figure 4: Confidence intervals of Llama3- 8B va. Mistral-7B for various sample size
6 Discussions
In this paper, we proposed model-free and nuisance-free approach for quantitatively comparing generative models with statistical confidence. First, we propose focusing on the relative performance gap (relative score) between two generative models, rather than evaluating their absolute performances individually. Second, we developed an unbiased estimator for the relative score that achieves parametric convergence rates and is asymptotically normal, enabling rigorous inference. Third, on simulated datasets with known ground truth, our method effectively controls type error while achieving comparable or superior power, whereas existing metrics often exhibit. near-zero coverage; when applied to generative models real image text datasets, approach yields statistically confident conclusions consistent with existing metrics but with uncertainty quantification.
The statistical significance under the proposed relative-Ki metric should not be over-interpreted. The com- parison only concerns whether the distribution of one generative model is closer to the true data distribution doesn't directly imply
than the distribution of another generative model in terms of the KL-divergence. other properties such as safety. When safety and harmful behaviors are the main concerns, specific metrics should be designed, and our method can serve complementary metric.
Acknowledgement
Portions of this work was performed on High Performance Computing resources provided by the Advanced Research Computing Services team at NJIT. Han Su was supported by the Doctoral Student Program of the Young S&T Talents Cultivation Project, CAST.
12

---

Published in Transactions on Machine Learning Research MMarch/2020)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tgee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprinl arXiv:2303.0877 08774, 2023.
Ikuo Arizono and Hiroshi Ohta. A test for normality based on kullback leibler information. Statistician, 43(1):20-22) 1989.
The American
Dmiiry I Belov and Ronald D Armstrong. Distributions of the kullback leibler divergence with applications. British Journal of Mathematical and Statistical Psychology, 64(2):291-30) 2011.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. Tn International Conference on Learning Representations (TCIRR) 2018.
Wacha Bounliphone, Eugene Belilovsky, Matthew B. Blaschko, Ioannis Autonoglou, and Arthur Gretton. A test relative similarity for model selection in generative models. In International Conference on Learning Representations (ICLR), 2016.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenand. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them. In Proceedings of the JEEE/CIS /CF conference on computer vision and pattern recognition, pp. 6070-6079, 2020.
Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. A kernel test of goodness of fit. In Proceedings of the 33rd International Conference on Machine Learning (ICML). PMI.R, 2016.
Eustasio Del Barrio and Jean-Michel Loubes. Central limit theorems for empirical transportation cost in general dimension. The Annals of Probability, 47(2)9926-951, 2019.
Laurent Dinh, Jascha Sohl-Diekstein, and Samy Bengio. Density estimation using Real NVP. ar preprinl arXiv:1605.0888 2016.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:207..2188,, 2024.
Lutz Dümbgen. On nondifferentiable functions and the bootstrap. Probability Theory and Related Fields, 95(1):125-140, 1993.
Jianqing Fan, Zhipeng Lou, Weichen Wang, and Mengxin Yu. Ranking inferences based on the top choice of multiway comparisons. Journal of the American Statisticul Association, pp. 1 1-14, 2024.
Jack Gallifant, Amelia Fiske, Yulia A Levites Strekalova, Juan S Osorio-Valencia Valencia, Rachael Parke, Rogers Mwavu, Nicole Martinez, Judy Wawira Gichoya, Marzyeh Ghassemi, Dina Demner-Fushman, et al. Peer review gpt-4 technical report and systems card. PLOS digital health, 3(10=4000 2024.
Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Peter Hall. The Bootstrap and Edgeworth Erpansion. Springer Science & Business Mcdia, 2013.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in newral infor- mation processing syslems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
13

---

Published in Transactions on Machine Learning Research (March/20P)
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statislics-Simalaliinn and Compulation, 18(3):1059-100 1076, 1989.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2024.
Albert Q Jiang, A Sablayrolles, A Mensch, C BBafford, Singh Chaplot, Ddl Casas, Bressand, G Lengyel, G Lample, I. Saulnier, et al. Mistral 7b. arxiv. arxiv preprint arXiv:2310. 0.06825, 10, 2023.
Marco Jiralcrspong et al. Feature likelihood divergence: Evaluating the generalization of generative models using samples. In Advances in Neural Information Processing Systems, volume 36, pp. 23005 23110, 2023.
Heishiro Kanagawa, Qiang Liu, Bharath K. Sriperumbudur, and Arthur Gretton. A kernel stein test for com- paring latent variable models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 85(3):986-1011 2023.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. Tn Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 8110 8119, 2020.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiw:1312.6114 2013.
Diederik P Kingma. Adam: A mothod for stochastic optimization. arXiv preprint 14 2014.
Arun Kumar Kuchibhotla, Sivaraman Balakrishnan, and Larry Wasserman. The hule: confidence regions from convex hulls. Journal of the Royal Statistical Sociely Series B: Statistical Methodology, 86(3):586 622, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXie preprinl Xiv: 12. 19437, 2024.
Qiang Liu, Jason Lee, and Michael I. Jordan. A kernelized stein discrepancy for goodness-of-fit tests. In Proceedings of the 33rd International Conference on Machine Learning (ICML). PMLR, 2016.
Ziwci Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on compuler vision, pp. 3730 3738, 2015.
Jerzy Neyman and Egon S. Pearson. On the problem of the most efficient tests of statistical hypotheses.
Philosophical Transactions of the Royal Society of London. Series A, 231:289 337, 1933.
Victor M Panaretos and Yoav Zemel. Statistical aspects of wasserstein distances. Annual review of statistics and its application, 6(1):405-431. 2019.
Dimitris N Politis and Joseph P Romano. Large sample confidence regions based subsamples under minimal assumptions. The Annals of Statistics, pp. 2031-2050, 1994.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International con- ference on machine learning, pp. 1530 1538. PMLR, 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International Con-
ference on Learning Representations, 2021.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neurra information processing systems, 32, 2019.
14

---

Published in Transactions on Machine Learning Research Mlarch/2020)
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguclov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1 9, 2015.
Shashanka Ubaru, Jie Chen, and Yousef Saad. Fast estimation of tr(((a)) via stochastic lanczos quadrature. SIAM Journal on Matriz Analysis and Applications, 38(4)10776-1099, 2017.
Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukeuoglu, et al. Wavenet: A generative model for raw audio. arXiv preprinl ar 1rx: 03499, 12, 2016a.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pp.
PMLR, 2016b.
Aad W Van der Vaart. Asymplolic slatislics, volume 3. Cambridge university press, 2000.
Cédric Villani el al. Oplimal trunsport: old and new, volume 338. Springer, 2009.
Quang H. Vuong. Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57 (2):307-336) 1989.
Samuel S. Wilks. The large-sample distribution of the likelihood ratio for testing composite hypotheses.
Annals of Mathematical Statistics, 9(1):60-62 62, 1938.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint ar Xrx:22200 01068, 2022.
Puning Zhao and Lifeng Lai. Minimax optimal estimation of KL divergence for continuous distributions. IEEE Transactions on Information Theory, 66(12):7787-788 7811, 2020.
15

---

Published in Transactions on Machine Learning Research Mlarch/20200
Appendix
Overview. The appendix is organized as follows. Section A reviews related literature on distance measures between distributions and quantitative evaluation metrics for image generative models. Section [ provides theoretical proofs of the main results. Section [ tion D presents additional empirical results and further descriptions of the simulation setups introduced the main text.
outlines computational and implementation details. Sec-
A Literature
A.1 Dissimilarity metrics between distributions
Quantitative evaluation of generative models typically involves compuling some distance between probability distributions. In this section, we review commonly used dissimilarity metrics between distributions and evaluation metrics of generative models.
A.1.1 f-Divergence
For a convex function f [0, +o0) + (-o0, too] such that f(r) is finite for all x > 0, f(1) = 0, and f(0) = lim;-o+ f(t), the f-divergence of P from Q is given by
(PI(ii)) /
p(v)
Pi (1)dy
(1(w)
KL-divergence is special case of f-divergence with f(x) log(z).
Estimating f-divergence typically requires the estimation of two densitics, and pi, where the estimation of exposes us to the curse of dimensionality. For the associated relative score, the convenient cancellation that simplifies computation is specific to the KT, divergence (Proposition 3.1). Consequently, for divergences other than KL, estimating the density becomes unavoidable, and the estimation of both absolute and relative f-divergences no longer achieves the parametric rate.
A.1.2 Wasserstein-p Distance
Wasserstein-p distance (Villani et al. 2009 is defined as
W (F P.) = inf
vET(P,F)) (ag)) [||x - =1
B1/P
where Pi) is the set of couplings between IP and P..
We next discuss the challenges of performing inference for evaluation methods based on the Wasserstein-p distance. First, Del Barrio Loubes (2019) shows thai
Vn W? [P. PI...) E[W (P...).)))) a AN N (0, (0,
where the asymptotic variance (IP P2)) can be estimated consistently using a plug-in estimator. The issue is that the center (P,, PP....] different from the desired (P, ([P.,P., PP1), and the gap between (P IP,..)) and (P, (Pi) scales as n -1/d (Villani et al. 2009 Second, for the relative Wasserstein-2 distance W (I,, Pi), (P PI.)) (P.. P2n)) is required.
remains unclear whether W. ) and W (P,,. P2..)) are asymptotically jointly Gaussian, as well as what the exact form of their covariance matrix 1 Third, subsampling methods (Dümbgen 1993) can be employed for conducting inference for Wasserstein-p distances; however, subsampling is computationally expensive, especially given that the Wasserstein-p distance already difficult compute. For comprehensive review of these issucs, see Panaretos & Zemel (2019).
the joint asymptotic distribution
10CTT results of general Wasserstein-p distance are largely unknown.
1he off-diagonal values the covariance matrix non-zero because both W3 P1.)) and WP P2.n) depend Pp-
16

---

Published in Transactions on Machine Learning Research MMarch/2020)
A.2 Existing evaluation metrics of generative models for image
We describe the inception score (IS) (Salimans el al. 2016) and the Frechet Inception Distance (FID) (Heusel et al., (017), two most commonly used quantitative scores for generative models. We note that these metrics were originally designed for training GANs rather than for model evaluation, with more emphasis placed on computational efficiency than on statistical rigor.
IS evaluates the quality of generative model by applying separate, pretrained image classification model to batch of images generated by the model. IS is maximized when the classifier confidently predicts single label for each image, or when the predictions are evenly distributed across all possible labels. The quality of IS depends heavily on the quality of the classifier (if the classifier consistently outputs a single label for all images, the IS becomes uninformative). Another disadvantage is that IS does not compare generated images to test images.
FID compares the distribution between the distribution of test images and that of generated images. Math- ematically, FID is defined as the Wasserstein-2 distance between the two distributions. However, the Wasserstein-2 distance is computationally expensive for random vectors, except for multivariate Gaussians. To approximate the FID, the default approach involves two steps: (1) mapping the real and generated images to Rd scparately by passing them through the final layer of an image classifier to extract essential features; (2) fitting multivariate Gaussian distributions to the transformed data in R and computing the W distance between these multivariate Gaussians. The approximation accuracy depends on how well the transformation to Rd captures the data characteristics and the quality of the multivariate Gaussians fit to the transformed data (the covariance matrix is typically not diagonal and the estimation involves (d2) elements). Chong& & Forsyth (2020) shows that FTD could be significantly biased in finite sample.
B Proof
B.1 Proof of results in Section 3
Proof of Proposition 4 For simplicity, use Pi instead of IP,, pi instead of Pı, and similarly for IP2 and p2. For any densitics p(z), (I)),, P2(x), we define
h(p.P..2) == / (i(e) f p(x)
(22(x))
(p(a)) p(z)dx.
p(x)
For any 6(z) such that f f(z)dr 0, and such that p(z) + tō(x)= 2 0, by the calculus of variations and 4
2 hpp + 16, pl, p1,,2))
at
(1()) p1(x)
(2((x)
P2(xr) pp(x))
8'
f
((x)p(x)dr
p(x))
p((x)
p(x)
(1()
p2(x)
6(z)de
(14)
p(x)
p(x)
(p((a)
(x)
Pi
(22(1) P2(x) p(x)
5(x) + l'
p(a) )
((x)
p(x)
p(x)
p1(2))
(palee ((z)dz.
+ f
8(x))
p(x)
We let f1(x) = f(er), then fi (log(z))
= f(x) and f((log(()) (1/2) f((x), which implies f(loo(()) =
(x). We replace f(x) by fi(z) in Eq. 14
f
(-ff fi log (())
fi log (((())
p(x)
p(x)
(15)
(2((z)) p(x)
))
P2 (2()x)
p(x) )) ((x)dr.
+ +
log
fi log
17

---

Published in Transactions on Machine Learning Research (March/20) /2026)
Since Equation 15 is true for arbitrary ((x) such that [(x) 0, then we let fa(x) -f((x) fi(z) and have
((1 (2), = (log(p((x)) - log(p(z))) - la (log(p2(<)) - log(p())) + C, Va.
(16)
for some constant E R. We assume C = 0, otherwise, we replace (2() by
C.
Note that f2(0) = - (0) + f1(0) - +
+ f(1) = -((0). We let fa(x) = f2(2) + ((), then f²(0)
and g(pi (z). (2))) /a (log(p) (x)) log(n(=))) (1og(p2(x)) log(p(x)))) Take p(œ) = pi (x) in Eq.
16
(fr( = fa(0) fa (log(12(x)) - log(n1 (2))) fa (log(12(x)) log(e) (x)))
(17)
We let a = (og(p((1)) - (og(p((x)) b (og(p(x)) log(p2(z)), then by 17
fa(a + b) - fb(b) = fs(a).
(18)
Since f C²2, then fs is continuous. By the result of Cauchy's s functional equation, there exists e, R
such that
fa(z) = cx d, Vr.
(19)
By the definition of fa(z) and 19 we have f2(x) = fs(()) - f((0) = cr + d' for some d' E R. By the definition of [2(x),
- +((x)+ + ct + d'.
This is standard ODE problem, and we multiply both sides by e = get
fe=*f(( cre + d'e
f((z) -crre = d'e +d"
f1(2) ac + Br +
where d", 3, 0 E R. By the definition of fi (x), f(x) = ax+ 3 3log(z)+ +0. Note that f(1) = 0, which implies a + 0 = 0. Note that [(((dP) a ad = 0, therefore, the divergence with f(x) = aæ + 3 log(z) + 0 is the same as that of 3 log(z). Since fx) is convex, we have 3 < 0, and we finish the proof.
B.2 Proof of results in Section 3.1
Proof of Proposilion Proposition follows from the linearity of expectation and the fact that the test
data Y ~ P.
Proof of Theorem 3.1 Recall that Y, are iidd. sampled from P. Theorem 3.1 an application of the Lindeberg-T.ew central limit theorem to i.i.d. random variables log(A1(1)) (Y)))
Proof of Covollary 3. When 0 < V < xc, by the law of large numbers of i.i.d. random variables, the empirical variance V of (as((1(()))) log(()(()))) converges to V in probability. We further combine Shutsky's 's theorem and Corollary 3.1 arrive Theorem 3.1
Proof of Corollary lary 3.1
3.2
Corollary
3.2 comes from the definition of convergence in distribution and Corol-
B.3 Proof of results in Section and additional discussions on EEs
The following assumptions ensure the existence of EEs of Z-statistics
Assumption B.1. Suppose the relative scores X have high-order moments, E|X|" < <0.
Assumption B.2. Suppose that the distribution F of relative scores admits absolutely continuous compo- ments, that is, lim + 11x (1) < 1.
18

---

Published in Transactions on Machine Learning Research (March/200)
Proof of Theorem 4.1. Under Assumptions B.1 and B.2. the EEs of Zn can be conducted in three steps: (1) Calculation of cumulants; (2) High-order expansion of c.f. 1/z, (4); (3) The Fourier inversion of c.f. 1/z, (t).
The definition and calculation cumulants. The definition of the cumulants of a random variable is related the polynomial expansion of its logarithmic characteristic function. Specifically, we first consider the logarithmic transformation of six (t), i.e.
Step 1:
log (1) = log + Aullaty
Σ Toly
where {uh21 are the moments of X, and {ah22 defined as the cumulants of X, obtained by expanding the log(1 + a) terms in the above equation via the Taylor series and matching the corresponding coefficients of polynomials. By simple calculations, the first four cumulants of X are defined as:
o2 = μ2 - 22 = 0²,
C1 μ.
C3 = μt 3 + 24 = E(X - 11)3 m3,
μ4 4uzh - 3 + 12p² μ2 6p = E(X 41² - -30 := my -30
where my and my are the third and fourth central moments of X, respectively, indicating the close relationship between cumulants and moments. Thus, six {at221 shown as follows:
be expressed exponential function of the cumulants
(t) = exp(log(xx()) = exp { itc + (it)
(it) 31
(it)* C
+ o +14)
2 02 +
+ 4!
Step 2: Expansion of the Characteristic Function. Similarly, the logarithmic characteristic function (x- - 11//a can also be expressed in terms of the cumulants (11)21
bby (t) of the standardized variable of Y:
(it)
(it)
(it)*
Ks + 0 ((t)
(t( exp itki +
R2
(20)
3!
4!
where 0, K2 = 1, K3 = m3/03 is the skewness of X, and ~4 = ma/o - 3 is the kurtosis of X.
Step 3: Inversion the Distribution Function. According to 20 the characteristic function v/2, (t)
of Zn in its expansion of CLT version
can be rewritten as a function of the cumulants as follows:
t
22 (t) = 4/
1
it
it
1
n
K3
+
R4
+
2
3!
V
4!
Vn
n2
22
(it)3
1
(it)a
exp
K44
2
31
Vn
4!
=
12 exp
K3
3
1
K44
(i))
VT
4!
-212
A3
(it)3
VLL
4!
1
164
(it)a
x (3!)2
Kig
(it) +
1 1
(21)
Furthermore, by inverting the distribution function via Fourier inversion, we can obtain the Edgeworth expansion results. Specifically, let (x)) denote the jth derivative of ((x). Noting that the characteristic function is the Fourier transformation of the distribution function, along with the formula of integration by
19

---

Published in Transactions on Machine Learning Research (March/200)
parts and some straightforward calculations, we obtain the following series of results:
e e2
ce d+(x)
2 (x)de
Φ(1) (I) 00
- -f c00)
-00
(-it) it
c c
(-it)
de
= (-it)- f eitm d(()))))))))
which indicates that (-it)e-- 12/2 is the Fourier transform of (x). Denoting the differential operator as D, we have
atr d(-D)/(x)) (itye-
(22)
Applying the differential operator repeatedly to ((x), can obtain that
(T(#)) = d(x) = Ho(x)((x), (2 = -xd(x) = - H1 (1(x)
(* = (x² 1))(()) = H2(z)0(z), D* -(x² -(8"-3r)d(()) -3r)d(r)
((2))))())
In sum up, we have
-H))))))))) j 0,1,2,
(23)
where {H,())} are the Hermite polynomials, satisfying the following properties: (1) (x) 120 are
orthogonal with respect to d(x), i.e,, 1 (z)=)((z))((())))) 0,j + k; (2) H2 is polynomial of degree j, and its parity (even/odd) matches the parity of the index j.
Naturally, according to 21
23 the characteristic function izz (t) of Zn can be viewed as the Fourier
transform of the following distribution:
((x) +
K3
(x))
+
K44
(-D) +
K2 (- ()) +o (
(
6
24n
72n
Ha(z)o(x) + Ho(z))(z) } fo
= +(x))
1/2
H2(z) n
(24)
24
72
This indicates that equation
24 represents the form of the distribution function of Zn- Combining with the discussion in Section 4.2 equation 10) can be further written as
(x) = ((x) n K/2
H₂ (z(r)((z) n
64 (z())(()) + Hb(x))(()) } + (n
(25)
24
72
20

---

Published in Transactions on Machine Learning Research (March/20) /2026)
25 is called the Edgeworth expansion for the Z-statistic. Furthermore, the result in for IR, that is,
25 holds uniformly
sup zER
(x) ((2) n 11/2
K3
H2(x)
n-
ka { 24
Hz(z))(z))
<3 Hs H=(0)o(0) o (n
6
72
which can be similarly derived following Theorem 2.1 in Hall 2013 and omit the details here. Thus, the proof of Theorem 4.1 is complete.
As the other side coins, the variance of relative scores log(β) (Y:)) - log((2(X))) usually unknown practice, one can consider the T-statisties T. (V ////ttt))-1/((( (8 (P2) - 5(PP, P2)), similar to Corollary 3.1 Formally, the EEs of Tú and its counterpart bias-corrected C1 can be derived, as shown in the following theorem.
Theorem B.1. The distribulion funclion of Tn salisfies
(x) (xx) + in-1// 1/2 K3 (2x² 1)d(() + n-
1 { kazll²² -3) ngg( +2x² -x2--) -3) xla² + +3) Φ(x) to(n
{
6
12
18
(a)
(26)
Furthermore, we can also derive the consistent. convergence result of Tn
(x) { (b(x) +((x))n=10 1/2 Ki (2x² 1)6(x) +n
1 12 kaaa((² - 3) ngnl(* + +2x -3) -000 3) a((5)
sup =
=0 (n
obtain the high-order expansion term of (z) based on the calculation of cumulants, the expansion and Fourier inversion of c.f. of T-statistics. The detailed proofs can be found in Section 2.6 in Hall 2013). and we omit the details here.
Proof of Theorem B.1 Similar to the proof of Theorem 4.1
Theorem B.1 also determins (1 or) leeel confidence interval of the relative score based on T-statistics
v
Clges (a)
BiV S(PP, - B2
(27)
m2
where 31, B2 satisfies
{ dG*(x) a, s.t. dG dG (82).
Both 12 and (27) derived by the limiting distributions of Theorems 4.1 and B.1 are valid Cls, shown in the following statement.
Corollary B.1. Under the Assumptions B.1 and B.2 for any E (0, 1),
P (8((1, P2) e ClEES(A)) = (()P), (a) + 1 a.
C Computation
Computing our relative score estimator involves the evaluation of the probability densitics (.(-) and f2(-) at each test point. We provide explicit details on how this computation conducted for generative models for image and text, respectively.
21

---

Published in Transactions on Machine Learning Research Mlarch/20200
C.1 Generative models for text
For models such auto-regressive language models using translormers, fi (-) can be computed directly through a forward pass. The computational cost of our method is comparable to that of standard metrics such as perplexity, which also requires evaluations of the log-likelihood.
C.2 Generative models for image
For generative models for images, samples are usually generated via an invertible transformation gr(-) 1- of a standard Gaussian variable (0, Ia,)- The density can be computed by mapping data points back to the latent space, where the latent embeddings follow a known multivariate normal distribution. Explicitly, let 91 be the inverse of 91, then by the change of variable formula,
(g)( = - 119 (v) d1 log(v2m) + log (1
(28)
Similarly, we obtain 28 for log(f2(y))- Then the estimator 8(P), 122) in takes the form
(119 (x ( + (d2 - d1) logloo22
neest
2
(29)
+ log (1J (oi (Y))) - log (1 1922 (Yi)))
(x)1))))
What remains to determine the inverse functions, 9.¹ and 92 well as the corresponding Jacobian determinants, 1 (Y:) and 1 (12l112()))) (Y:)). Below, discuss this computation over various generative models for images.
For normalizing flows (Rezende & Mohamed 2015 Dinh et al. 2016): the models constructed to allow both the inverse transformation and the leg-determinant of the Jacobian to be computed in closed form and evaluated efficiently.
In auto-encoders, the encoder effectively serves the role of inverse of the decoder, i.e., g_1
For diffusion models such as DDIM (Song et al. 2021 the forward process admits an inversion by solving the associated reverse-time ODE. Details in Section C.2.1
Remark 1 (Computation the Jacobian determinant). our experiments, we mainly consider image datasots with low resolutions. For this setting, we compute the exact Jacobian log determinants using In settings with very high image resolution or an exceptionally large number of test. data points, computation of the exact. Jacobian log determinants may be infeasible. We suggest approximating the log-determinant of the Jacobian using Stochastic Lanczos Quadrature (SLQ)
(Ubaru et al. 2017). Using the identity
log det(J)] log(J 1
tracce(lo((]) J)),
the trace term be estimated via stochastic trace estimation
Hutchinson 1989
m
(race(log(JTJ)) J))
Log(JTJJ) J)
with random probes 2; ~ NN0 f). Each quadratic form 2. (og(JT))=; J) can be computed using Lanezos itera- tions (Ubaru et al. 2017), which only require matrix vector products of the form T Jzz These products can implemented using Py Porch automatic differentiation (e.g., FORCH.FUNC.JJ and TORCI. FUNC. VJP), with computational cost comparable to standard backpropagation. With SLQ, the overall cost of evaluating (8
12For simplicity, consider generative models with cucoder-decoder structure and treat the encoder the inverse of the transformation g1(-) As shown Figure despite using approximate likelihoods, the resulting confidence intervals achieve coverage close nominal level. We provide further sensitivity analysis in Section D..3
22

---

Published in Transactions on Machine Learning Research March/2028
is the same order training the model for epoch the evaluation dataset. We further empirically as- sess the approximation error of SL.Q. Using the first test image of CIFAR-10 and the pretrained DDIM model ob- taining value of 16177.51. We then apply SLQ with single random probe and 10 Lanezos steps. Across 50 runs, SLQ yields mean estimate of approximately 16189.15 with standard deviation of 57.13. This demonstrates that STQQ provides reasonably accurate approximation, even with very small number of probes and iterations.
FUNCTIONAL..ACCIIN,, in Section 5.1.2 we compute log (Idet(Jo-)) 0)-)) exactly using Tai.
Remark 2 (Equivalent forms of Jacobian determinant). Since the Jacobian of the inverse function is the inverse matrix of the Jacobian of the original function, we have multiple equivalent forms for the term Io (i (Vi)) - (0g(()9a((0)) 1aalllo2'11 (Vi))) (V))))
in 22
log (1) 199 (YY)) - log (1 (92 (Xi)) (x2'())))
log (J (1g))1((i)) + log (10)
log (لمل 1602" (Y1))
log (1)9 oog 19 (X:)).
Explicitly, log (1/g I(xi)) computes the log determinant of the Jacobian matrix of the inverse function oT evaluated at the test data point Y; log composite function 91 92: evaluated at the latent embedding 92 (V,) of the test data point under the second generative model. In practice, either of these equivalent formulations can be chosen based on which Jacobian determinants can be computed more efficiently and accurately.
(V:)) computes the log determinant of the
In practice, the choice of form depends on whose Jacobian determinants can be computed more efficiently and accurately: the original (gi, 92), the inverse (9i 1 92 "), or the composite (gi g2)).
C.2.1 DDIM
The log of the Jacobian determinant can be expressed as the sum of S (total number of iterations in the sampling process) sub log-Jacobian determinants, with each term corresponding to the transformation in one iteration. each iteration, the transformation is given by,
2z-1 = aa-1
Fg
c (xx)
+ / - a=-1 E (xx),
(30)
a2
for a sequence of a E (0, 1). To compute the Jacobian determinant for this transformation, it suffices to compute the Jacobian of cg Since e/ is parameterized by U-Net architecture, its Jacobian can directly read from the trained U-Net.
(x)
The reverse of the sampling process, which encodes test image into latent noise, can be achieved by simulating the reverse of an ODE. In fact, DDIM can be considered as Euler method to solve ODEs. Specifically, the iterative formula can be represented as
I V
1
Ō,-1 Fe-l
xs
eø ca(2s. 8),
(31)
Os
We set yo = V a vs and p=:=
1,
a
Us-1 Us = (ps-1 (xss
(32)
In the limit of small steps, this equation becomes an ODE:
dy. co (T s)dp.-
(33)
23

---

Published in Transactions on Machine Learning Research Mlarch/20200
Then, the reversal of this ODE can be derived as follows:
ys+1 - Us = (Po+1 ())co(
(34)
which becomes,
1 0'++1
T V 0s
1
1 V
2s41
(Z(2)))) 8).
(35)
0a+1
Remark 3. Extending our evaluation method to generative models with non-deterministic reverse process, such as the Denoising Diffusion Probabilistic Models (DDPM) (Ho al. 2020), would be an intriguing direction for future research.
D Additional empirical results
D.1 Additional experiments for comparing CLT and EEs
In this section, we also design more complicated data generation process to compare the inference perfor- mances between our proposed EEs methods our proposed CLT (Z-statistics and T-statistics in Theorem 3.1 and Corollary 3.1 respectively) and EEs (Z-statistics and T-statisties in Theorem 4.1 and (B.1), respec- tively). Beyond the linear transformation of normal inpul (13), we also consider 5 types of input distribution of X, X2: normal, skewanormal, x2, Gamma and Beta distributions. The output data generation mech-
anism follows:
= A (0 B, Y2 = (A+cTa) o 9(2)) + B
(36)
where denotes the element-wiss wise product, 9(-) is the transformation function with certain smoothness to control the non-Gaussian degree of outputs, closer to the generative models in certain sense. Consider the following 8 forms of g(-) for evaluations: (1) g(X) = X. (2) 9(X) = =x², (3) 9X) = X 3/2 (4) g(X) =
sin(2 (5) x(X) = lXgit(X), (6) ((X) = a(X) with sigmoid transformation (=))= =
1/(1+e==)) (7) oX)) (X)=
ReLu(X)= = max X), (8) g(X)= = o(A o (AA o (X) + B) + B) with multilayer sigmoid transformation.
Other settings follow the simulated model 13 For each setting, we evaluate the performance of each method based on the following metrics: (1) empirical CI coverage rate, (2) empirical statistical power, and
(3) empirical CI length.
The results of Cls based on T-statisties 27 with unknown variance for the linear transformation and the sigmoid transformation are presented, respectively, in Figures 5 and 6 In Figures 5 and 6 the plots in the left column show the results for the EEs method, while the plots in the right column show the results for the CLT method. Each row corresponds to one of the three inference metrics. In all Figures, the a-axis represents the ground truth value = EEg[log 1 log p2), and dashed line marks the nominal 90% coverage level. Figures and G show that the Cls based on the EEs method not only can coverage the true interested parameter relative score, but also not sacrifice the testing capability and the length of CIs, yielding better trade-off between type I error and type IT error of statistical inference.
The additional simulation results of T-statistics under other transformations and those of Z-statistics are presented Figures 20 respectively.
intervals are asymptotically valid. The experiment results show that when the sample size is large (n 2 50), and the output distribution has normal tail, intervals con- structed from two methods almost the same; CLT-based methods would be easier to use and understand. When sample sizes are small (n < 50), the output distribution has heavy tail, non-negligible skewness and kurtosis, CLT-based intervals could have lower coverage rates. When the valid coverage rate (or type error) is the main concern, Delgeworth-expaasionn-beee is the main concern and slightly miscoverage acceptable, CLT-based methods will be preferred. Overall, we recommend the users to choose the CLT--bsodd statistics and intervals when the sample size n 2 50 and the output distribution has a normal tail; and choose the Eddgeworth-expassion based ones when S 50 the put distribution is far away from normal.
Both CLT--asod and Rigewwrth--xxpaasion--eeee
methods should be preferred. If the interval length
24

---

Published in Transactions on Machine Learning Research March/2028
FEs: Coverage Rate of 90% CI
CIT: Coverage Rate of 90% CI
1.0
Coveroign Rate
Coverage
0.6
0.6
0.4-
Romal
0.
Mermal
0.2
0.
Bets
0.0
0.3
0.0
0.1
0.3
Edllogp:
logpsl
Fallog
Pog
EEs: Power
CLl: Power
1.0
1.0
Normal
0.8
Power
0.6
Bets
Power
0.4
0.4
0.0
0.1
0.3
0.4
0.0
0.3
0.3
Egllogp:
logpps
Zollog
log
Length
Length
2.0
2.0
Nerral
San-ioma
1.5
1.5
Celegh
lergethh
1.0
0.5
0.5
0.0
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.1
Eglloggs
logps]
Eollogo:
log
Figure 5: Coverage rate, power and length of CTs constructed by EEs and CLT of T-statistics under linear transformation.
FEs: Coverage Rate of 90% CI
CIT: Coverage Rate of 90% CI
1.0
1.0
Coverage
0.6
Coverage
0.6
0.4
0.4
0.2
0.2
Data
Data
0.0
0.0
0.2
0.4
Fallogp: log.s1
Fallogo:
lag
Power
Power
1.0
1.0
0.8-
0.6
Power
0.6
0.4
0.4
Kornii
Socp-ommal
Stawwrrrmm
y?
Gamma Lata
Cammra
Bets
0.0
0.0
0.2
0.4
0.5
0.5
0.0
0.0
0.2
0.4
0.6
0.8
Fillogps
log
Fallog log pol
EEs: Length
Length
0.30
0.30
Sunenomal
(ength
0.20
Camma
lengghh
0.20
Beta
Nats
0.10
0.05
0.05
0.0
0.2
0.4
0.6
0.5
0.0
0.2
0.4
0.6
0.0
Eglloggs
log,px1
2ollogo;
log
Figure 6: Coverage rate, power and length of CIs constructed by EEs and CLT of T-statistics under sigmoid transformation.
25

---

Published in Transactions on Machine Learning Research March/2028
FEs: Coverage Rate of 90% CI
CUT Coverage Rate of 90% CI
1.0
1.0
Coverogge Rato
Coverege
0.6
0.6
0.4
0.4
Now
0.2
0.2
y
0.0
0.0
tellogp- logps1
zallogoy log
EEs: Power
Power
1.0
1.0
Power
0.6
Power
0.6
0.4
0.4
Harnd
blow mp
y
0.0
0.0
Lollogp: loggs1
Zollogo log
Length
Length
25
20
length
CI loroghh
Beta
Beto
10
30
A
Edllogp: log
Eollogos log
Figure 7: Coverage rate, power and length of CTs constructed by EEs and CLT of T-statistics under trans- formation g(X) X².
FEs: Coverage Rate of 90% CI
CIT: Coverage Rate of 90% CI
1.0
1.0
Coveragee Rata
Coverage
0.6
0.6
0.4
0.4
0.2
Deta
0.0
0,50
0.75
1.00
0.0
0.50
0.75
1.00
Fallopp:
log.b)
Falloge:
logos)
Power
Power
1.0
1.0
0.8-
0.6
0.4
Shanoramm
Normal
0.4
Sonn-ssmal
Gamma
Beta
Bata
0.0
0.00
0.25
0,50
0.75
1.00
0.0
0.00
0.25
0.50
0.75
1.00
F,llog
log.p)
Fallogp:
log Pr]
Length
Length
0.8
Gan--rama
Sonn-comal
0.6
0.6
(ength
Carria
longghh
Gamma tata
0.4
0.2
0.2
0.0
0.00
0.25
0.50
0.75
1.00
0.20
0.25
0.50
0.75
1.00
Kodlogg:
logp:1
Zallogoy
log
Figure 8: Coverago ratc, power and length of CIs constructed by EEs and CLT of T-statistics under trans- formation g(X) sin(2x
26

---

Published in Transactions on Machine Learning Research March/2028
ECs: Coverage Rate of 90% CI
CLT: Coverage Rate of 90%
1.0
Covenge
0.0
Coverage
0.6
C.-
Worrd
Norma
0.2
C.0
Eellogp
logps1
Callog pol
1.0
bes: Power
1.
CLI: Power
Struncmaa
C.6-
Power
C.4
DA
0,0
0.2
6,4
0.0
0.0
0.2
hglloge:
logps]
Bollog Psi
Length
Length
2.5
2.5
Verral
2.0
a"
2.0
langeh
1.0
1.0
6.5-
0.
0.
0.2
0.4
0.6
0.0
0.2
0.4
0.6
à = Follogp: - logps]
6
Fallog
Pog
Figure 9: Coverage rate, power and length of CTs constructed by EEs and CLT of T-statistics under ReLU transformation g(X) ReLu(XX ma
FEs: Coverage Rate of 90% CI
CIT: Coverage Rate of 90% CI
1.0
1.0
Coverage
0.6
Coveraige
0.6
0.4
0.4
0.2
Deta
0.0
0.0
Fallogp:
log.s)
Fallogo: logos!
Power
Power
1.0
1.0
0.8
0.8
0.8
0.4
0.4
horrl
Garraa
0.0
0.0
Follogpi log
Fallogp: log
Length
Length
Shan-rramal
length
8-
CI longth
Bera
Beta
2-
2
-2
-2 log pxl
Egdlogp: log
Zulloge:
Figure 10: Coverage rate, power and length of CIs constructod by EEs and CLT of T-statistics under Logit transformation g(X) lxg.
27

---

Published in Transactions on Machine Learning Research Mlarch/20200
FEs: Coverage Rate of 90% CI
CU: Coverage Rate of 90% CI
1.0
Coverege
0.6
Coveragge
0.6
0.4
0.4
0.2
0.2
Beta
0.0
-15
0.0
0.0
-0.5
tallogp: logps1
zallogoy log
EES: Power
Power
1.0
1.0
Power
0.6
0.4
0.4
Sonnetomaa
Gamma
Beto
0.0
-2.0
-1.5
-0.5
0.0
0.0
-2.0
-1.5
-1.0
-05
Eollogp:
log
Zollog
log
Length
Length
10
Celgghhh
CI lamgthh
Beta
Reto
-2.0
-1.5
-1.0
-0.
0.0
-1.0
-1.
-1.0
-0.
00
Edllog - log
Eallogo: log
Figure 11: Coverage rate, power and length of CTs constructed by EEs and CLT of T-statistics under transformation g(X) = |x/3/2
FEs: Coverage Rate of 90% CI
CIT: Coverage Rate of 90% CI
1.0
1.0
Coveraage Rata
Coverage
0.6
0.6
0.4
0.4
0.2
Deta
0.0
1.0
06
-02
0.0
0.0
-1.0
0.6
002
Fallopp:
logpsl
Fallogo:
log
Power
Power
1.0
1.0
0.8
0.4
Norml
0.4
Shan-ram
Senn-comaa
comra
Gamna
Beta
Eata
0.0
-1.0
-0.5
-06
-04
-02
0.0
0.0
-2.0
-0.8
-0.6
-0.4
-022
0.
F.llogp
log.s1
Fallogp: log
Length
Length
2.5
2.5
2.0
nameenamal
2.0
Gomme
lenggth
Gamma
Beta
Betn
1.0
1.0
0.5
-1.3
-0.5
-0.6
-0A
-022
0.0
-1.0
-0.0
-0.8
-0.4
-0.2 0.0
kgllogp:
log,211
Zallog
0;
log
Figure 12: Coverage rate, power and length of CIs constructod by EEs and CLT of T-statistics under multilayer sigmoid transformation g(X) aaAAolAA BB) B).
28

---

Published in Transactions on Machine Learning Research March/2028
FEs: Coverage Rate of 90% CI
CIT: Coverage Rate of 90% CI
1.0
Coveroign Rate
Coverage
0.6
0.6
0.4-
Romal
64
0.2
0.
Bets
0.0
0.3
0.0
0.1
0.3
Edllogps
logpsl
Fallog
Pog
EES: Power
CLl: Power
1.0
10
0.8
0.8
0.6
Power
0.4
0.4
0.0
0.1
0.3
0.4
0.
0.3
0.3
Egllogp:
logpss
Zollog
log
Length
Length
2.0
2.0
Nerral
San-roma
1.5
1.5
Cerggh
lergeth
1.0
0.5
0.0
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.1
Eglloggs
logps]
Eollogo:
log
Figure 13: Coverage rate, power and length of CTs constructed by EEs and CLT of Z-statistics under linear transformation.
EEs: Coverage Rate of 90%
CIT: Coverage Rate of 90%
1.0
1.0
Coveragee Rata
Coverage
0.6
0.6
0.4
0.4
0.2
Deta
0.0
0.0
Fallog log
Fullog -lgg
Power
CIT: Power
1.0
1.0
0.8-
0.6
0.4
0.4
Kornii
Secp-nomal
Stawwrrmm
y?
Gamma
Camrra
Eata
Bets
0.0
0.0
0.0
0.2
0.4
0.5
0.5
0.0
0.2
0.6
0.8
5
Fallogp
log
Fullog pi Ing
Length
Length
0.4
Gaae-nama
Sonn-comal
0.3
0.1
(ength
Carrna
longth
Gamna
Bats
tath
0.2
0.1
0.1
0.0
0.0
0.2
0.4
U.S
0.5
0.0
0.2
0.4
0.6 log
0.0
5
Zallogpy
log.s11
à -
Fellog Pi
Figure 14: Coverage ratc, power and length of CIs constructod by EEs and CLT of Z-statistics under sigmoid transformation.
29

---

Published in Transactions on Machine Learning Research March/2028
FEs: Coverage Rate of 90% CI
CUT Coverage Rate of 90% CI
1.0
1.0
Coveraige Rato
Coverege
0.6
0.6
0.4
0.4
0.2
0.2
Beta
Beta
0.0
0.0
tallogp:
logps1
zallogoy log
EEs: Power
Power
1.0
1.
Nomal
Memml
Ohl
Carrra
Power
0.6
Bess
Power
Eata
0.4
0.4
0.0
0.0
Lollogp: loggss
Zollog log
Length
Length
length
CI laroeth
Beta
Beto
A Edllog
logps]
Eollogos log
Figure 15: Coverage rate, power and length of CTs constructed by EEs and CLT of Z-statistics under transformation g(X) x
FEs: Coverage Rate of 90% CI
CIT: Coverage Rate of 90% CI
1.0
1.0
Coveragge Rata
Coverage
0.6
0.6
0.4
0.4
0.2
Deta
0.0
0,50
0.75
1.00
0.0
0.75
1.00
Fallopp:
log.b)
Falloge: logos)
Power
Power
1.0
1.0
0.8-
0.6
0.4
Shanoramm
Normal
0.4
Sonn-smmal
Gamma
Beta
Bata
0.0
0.00
0.25
0,50
0.75
1.00
0.0
0.00
0.25
0.50
0.75
1.00
F,
log.p)
Fillogp:
log Pr]
Length
Length
0.8
Gan--rama
Sonn-comal
0.6
0.6
(ength
Carria
longgth
Gamma tats
0.4
0.2
0.2
0.0
0.00
0.25
0.50
0.75
1.00
1.00 0.20
0.25
0.50
0.75
kollogp:
logp21
Zallogoy logo:!
Figure 16: Coverage rate, power and length of Cls constructod by EEs and CLT of Z-statistics under transformation g(X) xi).
30

---

Published in Transactions on Machine Learning Research March/2020
ECx: Coverage Rate of 90% CI
CLT: Coverage Rate of 90%
1.0
Covenge
C.0
Coverage
0.6
C.-
Womrel
Norma
C.2
C.0
UC
Eellogp
logps1
Callog log pul
bis: Power
CLI: Power
1.0
1.c
Strunemaa
C.6-
Power
06
C.4
D.
6,0
0.2
6,4
0.0
0.0
04 bollog
0.2
hglloge:
logps]
6
Ps
Length
Length
2.0
3.0
Scrral
Sonveerral
a"
lengeh
2.0
langeth
2.0
1.5
2.5
0.5-
0.5
0.
0.0
0.
0.2
0.4
0.6
0.0
0.2
0.4
0.6
à = Fallogp: - logps]
6
Fallog
Pog
Figure 17: Coverage rate, power and length of CTs constructed by EEs and CLT of Z-statistics under ReLU transformation g(X) ReLu(X)= max(0, X).
FFs: Coverage Rate of 90% CI
CIT. Coverage Rate of 90% CI
1.0
1.0
Coverage
0.6-
Coverage
0.6
0.4
0.4
0.2
0.2
Data
Data
0.0
0.0
Fullogp: log
Fallog logos]
EEs: Power
CLT: Power
1.0
1.0
0.8
Power
0.5
Power
0.4
0.4
0.0
0.0
Follog log
Fallog log
Length
Length
12-
10:
telgghh
8-
Bera
CI length
Beta
Eallog P:
1og
5
Zallog
log
Figure 18: Coverage rate, power and length of CIs constructod by EEs and CLT of Z-statistics under Logit transformation g(X) lx).
31

---

Published in Transactions on Machine Learning Research March/2028
FEs: Coverage Rate of 90% CI
CIT: Coverage Rate of 90% CI
1.0
Coveraign Rate
Coverege
0.6
0.6
0.4
Nomal
0.
0.2
02
Beta
0.0
-15
0.0
0.0
-05
Eallogp: logps1
zallogoi log
EES: Power
Power
1.0
1.0
0.8
Power
0.6
Power
0.4
0.4
0.0
-2.0
-1.5
-0.5
0.0
0.0
-2.0
-1.5
-1.0
-0.5
Eollogp:
logps]
Zollog::
log Pzl
Length
Length
10
Skme-nomm
Celegghh
CI largehh
Beta
Beto
2-
-2.0
-1.5
-1.0
-0.5
0.0
-2.0
-1.5
-1.0
-0.
0.0
Edllogp: log
Eollogo: log
Figure Coverage rate, power and length of CTs constructed by EEs and CLT of Z-statistics under transformation g(X) = |x/3/2.
FEs: Coverage Rate of 90% CI
CIT: Coverage Rate of 90% CI
1.0
1.0
Coveragee Rata
Coverage
0.6
0.6
0.4
0.4
0.2
Deta
0.0
1.0
06
-02
0.0
0.0
-1.0
0.6
002
Fallopp:
logpsl
Fallogo:
log
Power
Power
1.0
1.0
0.8
0.8
0.4
Nomal
0.4
Sernecomaa
comra
Gamna
Beta
Esta
0.0
-1.0
-0.5
-06
-04
-02
0.0
0.0
-1.0
-0.8
-0.6
-0.4
-022
0.
F.llogp:
log.s1
Fallogi: log Pg]
Length
Length
2.5
2.5
2.0
name-namaa
2.0
S0en
Gomme
tenggth
Gamma
Beta
Betn
1.0-
1.0
0.5
-1.3
-0.5
-0.6
-0A
-02
0.0
-1.0
-0.0
-0.8
-0.4
-0.2
0.0
Kollogp:
log,211
Zallog
0)
log
Figure 20: Coverage rate, power and length of Cls constructod by EEs and CLT of Z-statistics under multilayer sigmoid transformation g(X) alA@olA +B) B).
32

---

Published in Transactions on Machine Learning Research Mlarch/20200
D.1.1 Choices of Optimal Using EEs
Figures 22 and 21 compare the inference performances of EEs of T-statistics with unknown variance across different imput distributions of and different sample sizes n, and we use the CLT of T-statistics as com- petitor. We find that EEs successfully keep the coverage rate at the nominal level while CLT suffers bigger coverage errors over a, especially when the sample size is relatively small (like n 100). These further point out that the EEs can be used for early stopping and the optimal training sample determination in the inference of relative scores.
Normal: Coverage Rate 90%
Normal: CI Lengtn
0.
0,2
Coverage
0.8
0.7
0.0
10'
10'
10'
Coverage
Skew-nomaa Length
Langtth
0.2
t5
0.8
0.3
0.0
Coverage
Langth
9.
0.0
Ebs
Gamma: Coverage Rate 90%
Gamma. CI Longth
0.0
0.2
0.8
0.2
Beta: Coverage Rate of 90% CI
Beta: CI Length
0.
Length
0.2
68
Leve1 50
0.7
0.0
10'
10
10'
10'
Figure 21: Coverage rates and length of confidence intervals obtained by EEs 12 and CLT 7 across different distributions of X and sample sizes n using simulated data generated by (36 with sigmoid transformation, e 0.07.
D.2 Additional details for Section 5.1.1
We investigate why existing estimators fail provide valid confidence intervals by examining their distri- butions numerically. Figure 23 presents histograms of the estimators alongside the true values of relative KL divergence and relative W2 distance when 0.05. The results show that these estimators exhibit significant bias, making them unsuitable for reliable inference. This empirical observation is consistent with our discussion of the challenges in estimating KI. divergence and W2 distance (Section 3
We also provide more details on the estimated inverse learned by aulo-encoders used in the simulations of Section 5.1.1 Specifically, we consider the same data generation mechanism (13), and generate N = 5 10a samples from Y, (from 91) and Y2 (from 92) and train two auto-encoders to reconstruct the data. Both the encoder and decoder are fully connected neural networks with a single hidden layer containing 100 units. The auto-encoders are trained for 20 epochs using the Adam optimizer (Kingmal 2014). To enforce that the
33

---

Published in Transactions on Machine Learning Research MMarch/2020
Coverage
Length
1.00
Longeh
0.75
0.8
0.25
Snowinormal: Coverage Rate 90%
Skew normal: CI Length
1.00
0.50
0.81
0.25
0.00
x": Coverage Rate 90% CI
X" CI Length
1.00
0.0
Langee
0.50
0.8-
Lowelo 50
0.7
10
0.00
16
10
Germmnr Coverage Rate 90%
Gamme: CI Length
1:00
Longtn
0.75
Coverage
0.50
0.0
L.vela.00
0.7
19
10
0.00
16
15
10'
Longth
0.75
0.8
CD7
0.25
0.00
10F
Figure 22: Coverage rates and length of confidence intervals obtained by EES (22 and CLT 7 across different
distributions of X and sample sizes using simulated data generated by 36 with lincar transformation. = 0.07.
Histagram of W2 Difference Estimator
Histogram of KL Difference Estimator
Tue W2 Dfference
Trus KL Difforencs
0.25
Figure 23: Histogram of existing estimators for the W2 distance and KL divergence. The vertical line indicates the true value in the simulated example.
34

---

Published in Transactions on Machine Learning Research (March/200)
encoder maps the data to NoO, 1), introduce penalty term (L (O, Ia)), where j and E are the empirical mean and covariance matrix of the encoder outputs. We use the two learned encoders as gr and 92 respectively.
To assess the performance of our method, we again generate 7 = 1000 samples from Y, compute the estimator in 29 and construct the confidence intervals using (7). The coverage rates over 1000 repeated experiments are summarized in Figure | (denoted as "Ours (Auto Encoder))). The results show that our method achieves coverage rates close to the target level, even when using the learned inverse functions.
More generally, for the case where the generator g's inverse g-1 is not directly available, we can train an auto-encoder minimizing the reconstruction error using the data generated from and use the encoder as g To understand the effect of the approximation errors in g² we conduct a sensitivity analysis using the simulated data. We compose the encoder for Y2 with a perturbation function
h(X) = X a (1 5Z),
where ~ MO,, 1) a gaussian vector with the same shape as X, denote the elementwise product. controls the strength of the perturbation. We compute the likelihood of Y2 using the perturbed encoder and evaluate the coverage rate of our method. The results are given in Table In general, when the approximation error is small (in this case, 8 s 0.01), the coverage is not substantially affected, indicating that our approach is not highly sensitive to small approximation error.
Table 4: Coverage rate and bias under different perturbation levels
0.00
0.01
0.02
0.03
0.04
0.05
Coverage Rate
0.906
0.894
0.842
0.770
0.660
0.462
Bias
3.74x 10-3
4.79 10
7.57 x 10 3
1.38 x 10
1.94 10- -2
2.88 x 10
D.3 Additional details for Section 5.1.3
For wikilext data, there might exist potential non-iid structure on the datasets of language models, when considering that the dataset may contain multiple paragraphs from the same article, such as the WikiText datasot. In such cascs, the sequence of the input data {X,)221 (random variables) can be said to have a m-dependent structure: there exists a small integer m (ie.. the maximum of the number of paragraphs in each article), such that for any n 2 and k 2 m, holds that X,t* is independent of F, = (x.,....,, the o-fildd generated by {X In such m-dependent case, under two mild conditions: (C1) (X.) have uniformly bounded variance such that [mn\/33- "VaV'VVar X,) our proposed estimator of the relative score is still unbiased and asymptotically normal, and hence the statistical inference and uncertainty quantity based on the relative score still work. The corresponding
as - co, and (C2) m //),
theoretical results about the CLT and EEs for m-dependent sequences can be found in Van der Vaart (2000)
and Hall 20133..
We provide experiments comparing CLT-based intervals and EEs intervals. Figures 24 presents the pairwisc CIs comparison results of GPT2 vs. GPT2-Large.
D.4 Additional details for Section 5.2
For TriviaQA, we compute the likelihood for single answer per data point. Specifically, we use the answer "aawwwrr]]"" "[veerr[[vvwa valuu] and do not perform additional answer normalization or aggre-
string provided by example['answerr]) gation of probability mass across multiple acceptable answers. The likelihood is therefore evaluated on this single reference answer only.
In addtion, we provide experiments comparing CLT-based intervals and EEs intervals. Figures 25 and 26 present the pairwise CIs comparison results of LLMss: OPT-1.3B vs. Mistral-7B, and OPT-1.3B vs. Llama3- 8B. The results indicate that both of the Mistral-7B and the Llama3-8B models are significantly different from OPT-1.3B model, which makes sense because of the quite different memory sizes.
35

---

Published in Transactions on Machine Learning Research (March /2026
Confidence Intervals for EEs and CLT
CUT
Confidence intervals
19
100
100
500
800
1000
2185
Figure 24: Comparison between GPT2 and GPT2-Large on WikiText-2 with different sample size
Confidence Intervals for EEs and CLT
Confidercce
cu
10
20
30
50
100
300
500
800
1000
2000
5000
10000
11313
Figure 25: Cls of OPT-1.3B vs. Mistral-7B for various sample size n.
36

---

Published in Transactions on Machine Learning Research (March/200) /202))
Confidence Intervals for EEs and CLT
10
Confidence intervals
100
300
500
800
1000
2000
5000
10000
11313
Figure 26: CIs of Llama3-8B vs. OPT-1.3B for various sample size n.
D.5 Additional Experiments Generative Models for Image
We apply method to evaluate generative models the CelebA dataset (Liu al. 2015). Specifically, we trained a VAE model using the default settings from the GitHub repository and compared it with a pre-trained DDIM model (Song et al. 2021 with S = 20 denoising steps. The results are summarized in Table 5 Our results are consistent. with FID scores, but our method additionally provides statistical confidence in the comparisons.
Table Comparison of VAE model and DDIM model with S 20 denoising steps on CelebA data. Our confidence interval doesn't cover 0, indicating the DDIM model is significantly better than the VAE model.
Model M
FID
Ci of 6(pMM PDDIM..)
VAE
150.65
(-45912.80, -45690.22)
DDIM20
15.26
E Future Directions
We outline several promising directions for future research.
Extension to the comparison of multiple generative models. Our estimator 5 of the relative score and its asymptotic distribution characterization Thcorem 3.1 naturally extend to pairwise com- parisons of multiple generative models. Combined with Fan et al. (2024), we can identify the best-performing model and establish the full ranking of multiple generative models with statistical confidence.
Heterogeneity in relative performance. There may be significant heterogeneity the relative perfor- mance of generative models across test datasets, egg., model that excels generating cat images might perform poorly strengths of different generative models with statistical confidence, which can further guide the devel- opment of expert models that strategically leverage the strengths of individual models for improved performance.
images. In future works, aim to use method to identify the
hhttp:://giiuub..cm//ttiitit
37


Reviews:
[{"review_id": "PEtmU8rlQA", "reviewer": "Reviewer_vp9U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a likelihood-based framework to compare generative models with uncertainty quantification by using the difference in KL divergences to an unknown test distribution, defining a relative KL score. The key idea is that taking the difference cancels the intractable data entropy term, so the target reduces to an expectation under the test distribution of a known log-likelihood ratio. The estimator is simply the average per example log-likelihood difference, which enables standard asymptotic inference and a variance-based confidence interval. Empirically, the authors report improved coverage compared to KL and Wasserstein based estimators paired with resampling methods, and higher power than kernel based baselines such as MMD and KSD in simulations. They further present case studies on image models (DDIM variants, normalizing flows, and VAEs) on CIFAR-10, and on language models evaluated on WikiText2 and TriviaQA.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims are well supported by theoretical analysis. The appendix provides proofs (or standard derivations) for the key theoretical statements, including: the KL-specific uniqueness/cancellation result, unbiasedness of the proposed estimator, asymptotic normality via standard iid CLT arguments, and the Edgeworth expansion derived through cumulants and characteristic-function expansions under explicit moment and regularity assumptions. The appendix also discusses the unknown-variance (t-statistic) refinement and links it to classical results.\nHowever, the empirical evidence would be more convincing if the paper were more explicit about how likelihoods are computed for each model family in the reported experiments. In particular, the paper does not quantify how approximation error in likelihood evaluation (when exact likelihoods are not available or are computed approximately) propagates to bias/variance and ultimately affects confidence-interval coverage and hypothesis-testing validity. Clarifying the exact likelihood computation pipeline and providing a robustness/sensitivity analysis would strengthen the empirical support.", "audience_yn": "Yes", "audience_explanation": "This paper addresses a clear gap: most widely used generative-model metrics (e.g., FID, perplexity) are point estimates and do not provide uncertainty quantification. The proposed approach is simple and scalable, compute a held-out average log-likelihood ratio and attach confidence intervals, and it naturally extends to conditional models. As a result, it is directly relevant to modern conditional generators and language models, and provides a principled statistical perspective on generative model evaluation that many in TMLR’s audience would find useful.", "requested_changes": "# Critical\n1. Clarify and validate likelihood evaluation, especially for diffusion/DDIM models. Appendix C suggests DDIM likelihoods rely on per-step Jacobian determinants; the paper should clearly state what is actually computed in experiments and how it is made tractable.\n2. For each model family, explicitly specify whether the likelihood used is exact, a bound (e.g., ELBO), or an approximation, so the comparisons are consistent and interpretable.\n3. Analyze how likelihood approximation error affects inference, bias, variance, and CI coverage, ideally via a sensitivity analysis or targeted ablations.\n4. Provide practical guidance on when to use CLT-based intervals versus Edgeworth-expansion intervals (e.g., based on sample size and tail behavior), so users can apply the method safely.\n\n# To strengthen the work\n1. For language-model evaluation on datasets like WikiText, discuss potential non-iid structure (e.g., within-document dependence) and add guidance for multiple pairwise comparisons (e.g., FWER/FDR control).\n2. For TriviaQA, specify how answer normalization/aliasing is handled and whether probability mass is aggregated over multiple acceptable answers.\n3. Although the estimator is linear in the number of test points, likelihood evaluation can dominate the cost (and may be far from cheap for diffusion); report runtimes and computational details for all evaluated models and baselines.", "broader_impact_concerns": "A main concern is that statistical significance under the proposed relative-KL metric can be over-interpreted: improved likelihood does not necessarily imply better quality, usefulness, or safety, especially if rare harmful behaviors are underrepresented in the test set.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TZyQXnF4H3", "reviewer": "Reviewer_AUbY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a statistical framework for rigorously and quantitatively comparing two generative models. Existing evaluation metrics (e.g., FID and Wasserstein distance-based metrics) often lack uncertainty quantification, making it difficult to draw statistically grounded conclusions. The key contribution is an evaluation method tailored to the practical question of which of two generative models is better, equipped with confidence intervals and significance testing.　\n\nThe core idea is to estimate a KL-based relative score rather than absolute scores. Due to a cancellation property specific to KL divergence, nuisance terms depending on the unknown true distribution vanish, and the relative score can be written as an expectation of a test-data log-likelihood difference. This yields a simple unbiased sample-mean (first-order U-statistic) estimator with $\\sqrt{n}$-rate convergence and asymptotic normality, thereby enabling principled confidence intervals and statistical significance testing.\n\nSection 5 validates the proposed method via simulations and real-data experiments across modalities, including image generation (CIFAR-10) and text generation / conditional evaluation (WikiText-2, TriviaQA). The resulting rankings are broadly consistent with standard metrics (e.g., FID and perplexity) while additionally quantifying statistical significance, and the experiments suggest the proposed method provides better-calibrated inference than several baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "YES. \n\n**Theoretical evidence.** The KL-based relative score in Eq. (2) admits a key cancellation: the intractable entropy term induced by the unknown true distribution vanishes, so the target reduces to an expectation of the test-data log-likelihood difference as in Eq. (3). This paper further shows this cancellation is essentially specific to KL divergence (i.e., it does not generally hold for arbitrary $f$-divergences; Proposition 1). The resulting estimator is a simple sample mean (a first-order U-statistic), which is unbiased (Proposition 2) and asymptotically normal at $\\sqrt{n}$-rate (Theorem 3.1). This directly enables normal-approximation confidence intervals (Eq. (7)) with asymptotically correct coverage $1-\\alpha$ (Corollary 3.2). Section 4 additionally discusses practical extensions, including conditional generative models and small-sample settings.\n\n**Empirical evidence.** In Section 5.1.1, simulations evaluate both coverage and power, supporting that the proposed inference procedure is well-calibrated in controlled settings. Sections 5.1.2–5.1.3 further validate the method on real benchmarks for image and text/conditional generation, showing conclusions consistent with standard metrics (e.g., FID and perplexity) while additionally providing statistical significance statements via confidence intervals.", "audience_yn": "Yes", "audience_explanation": "Yes. \n\nMany researchers will have concerns about the lack of uncertainty quantification in generative model evaluation. I think this paper presents an effective approach to address that issue.", "requested_changes": "I found the paper well written and well organized, and I do not think it requires major revisions. That said, one additional experiment could further strengthen the empirical evidence.\n\n**Sample-size sensitivity on unconditional benchmarks (CIFAR-10, WikiText-2).** While the appendix investigates sample-size sensitivity in the conditional TriviaQA setting, a comparable analysis is missing for the unconditional benchmarks. It would be helpful to run the same type of subsampling study on CIFAR-10 and WikiText-2 and report how the CI width and the frequency of excluding zero change with the test-set size.", "broader_impact_concerns": "The proposed method aims to make the evaluation and comparison of generative models more rigorous and is not intended to directly enhance generative capabilities. If anything, it may be beneficial in helping prevent overstated claims or misleading comparisons. Accordingly, I do not see major social concerns in this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sVFPDrOzjP", "reviewer": "Reviewer_ayqR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "To address the \"lacking uncertain property\" of metrics like FID or kernel-based metrics, the paper proposes a relative KL divergence metric and extends the metric to conditional generative models. Theorems and experiments are presented.", "claims_and_evidence_yn": "No", "claims_explanation": "1. The paper challenges the \"lacking uncertain property\" of FID or other metrics. However, for readers even working on generative models, we cannot get the point \"uncertain property\". Why does the proposed metric (related KL divergence) formulate the uncertain property? Please refer to the section \"Requested changes\".\n\n2. From the experiments, it is somehow hard to find the superiority of the proposed metric. Better presentation is expected.", "audience_yn": "Yes", "audience_explanation": "The topic \"reasonable metric of generative models\" is very vital in the community. The community indeed needs better and more reasonable metrics.", "requested_changes": "1. The motivation. What is the \"lacking uncertain property\" of previous metrics? Can authors provide more thorough and convincing illustrations? I guess, authors may challenge the \"ground truth\" (i.e., the statistical properties of the training dataset) is statically calculated, which cannot reflect the property of the distribution. That is the flaw of the statically constructed dataset (e.g., ImageNet). If we check the calculation of FID, the mean $\\mu$ and std $\\sigma$ of high level Inception features of the training dataset are used. It is about static property, rather than \"lacking uncertain property\". Moreover, why does the proposed metric address the \"lacking uncertain property\" of previous metrics?\n\n2. The calculation of the proposed metric. In Eq. (3), why do authors claim that calculating the entropy of the training/real dataset $\\mathbb{P}$ is challenging? Do authors mean that the coverage of all samples in $\\mathbb{P}$ is challenging? In Eq. (3), how do authors calculate/approximate the remaining terms? I mean, the integration over $\\mathbb{P}$ is also challenging. In practice, we do not know the probability of $y\\sim \\mathbb{P}(y)$. In most cases, we use $\\frac{1}{N}$, where $N$ is the size of sampling.\n\n3. The results. From the presented results, I cannot see the potential value of the proposed metric.  Can the proposed metric guide a better generative model, e.g., Text-to-image models or Noise-to-image models? Figure 1 is expected to show more direct advantages of the proposed metric.", "broader_impact_concerns": "N/A", "additional_comments": "1. For the theorem, I do not rigorously check the correctness.\n\n2. Better presentations and illustrations of the motivation, the method are expected.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)