PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation
Authors: Pablo Lemos
Sammy Nasser Sharief
Nikolay Malkin
Salma Salhi
Connor Stone
Laurence Perreault-Levasseur
Yashar Hezaveh
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Models
Model Evaluation
Sample-based Metrics
Hypothesis Testing
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We propose PQMass, a novel two-sample test for assessing the quality of generative models.

Abstract: We propose a likelihood-free method for comparing two distributions given samples from each, with the goal of assessing the quality of generative models. The proposed approach, PQMass, provides a statistically rigorous method for assessing the performance of a single generative model or the comparison of multiple competing models. PQMass divides the sample space into non-overlapping regions and applies chi-squared tests to the number of data samples that fall within each region, giving a $p$-value that measures the probability that the bin counts derived from two sets of samples are drawn from the same multinomial distribution. PQMass does not depend on assumptions regarding the density of the true distribution, nor does it rely on training or fitting any auxiliary models. We evaluate PQMass on data of various modalities and dimensions, demonstrating its effectiveness in assessing the quality, novelty, and diversity of generated samples. We further show that PQMass scales well to moderately high-dimensional data and thus obviates the need for feature extraction in practical applications.

Published as a conference paper at ICLR 2025
PQMASS: PROBABILISTIC ASSESSMENT OF THE QUALITY OF GENERATIVE MODELS USING PROBABILITY MASS ESTIMATION
Pablo Lemos 1,2,3,4551 Sammy Sharief" 1,23f
Nikolay Malkin Salma Salhi
Connor Stone Laurence Perreault-Levassuur 122357
Yashar Hezaveh 1,2,5,,7
Mila Québec Artificial Intelligence Institute
Ciela Institute UUniversity of Edinburgh
Université de Montréal Trottier Space Institute
"SanboxAQ SCCA Flatiron Institute
These authors contributed equally.
ABSTRACT
We propose a likelihood-free method for comparing two distributions given samples from each, with the goal of assessing the quality of generative models. The pro- posed approach, PQMass, provides a statistically rigorous method for assessing the performance of single generative model or the comparison of multiple competing models. PQMass divides the sample space into non-overlapping regions and applies chi-squared tests to the number of data samples that fall within each region, giving a p-value that measures the probability that the bin counts derived from two sets of samples are drawn from the same multinomial distribution. POMass does not depend on assumptions regarding the density of the true distribution, nor does it rely training or fitting any auxiliary models. We evaluate PQMass on data of various modalities and dimensions, demonstrating its effectiveness in assessing the quality, novelty, and diversity of generated samples. We further show that PQMass scales well to moderately high-dimensional data and thus obviates the need for feature extraction in practical applications. Code: https://github.com/// MaatCCila--nttuute/OMM
INTRODUCTION
Generative modeling the task of inferring distribution given set of samples - is important and ubiquitous task in machine learning. Generative machine learning has witnessed the development of succession of methods for distribution approximation in high-dimensional spaces, including variational autoencoders (VAEs, Kingma & Welling, 2014), generative adversarial networks (GANs, Goodfellow al., 2014), normalizing flows (Rezende Mohamed, 2015), and score-based (diffusion) generative models (Ho et al., 2020). With advancements in generative models, evaluating their performance using rigorous, clearly defined metries and criteria has become increasingly essential.
The ability to distinguish between true and modeled distributions has become increasingly crucial, particularly in the context of AI safety across various applications. This is evident in concerns regarding data privacy (e.g., Somepalli et al., 2023; Hitaj et al., 2017) and the proliferation of Al- generated content across different domains of the internet (e.g., s Knott et al., 2024). As systems become more pervasive, accurately assessing the fidelity of generated distributions to their world counterparts is essential for ensuring the integrity and safety of AI applications.
In scientific applications, recent years have seen increased application of deep generative models. For example, they have been used as expressive plug-and-pla- priors (e.g., Song et al., 2022; Chung & Ye, 2022; Rozet & Louppe, 2023; Kawar et al., 2022; Dou & Song, 2024; Chung et al., 2023; Feng et al., 2023; Dia et al., 2023; Drozdova et al., 2024: Xue et al., 2023; Remy et al., 2023; Flöss al., 2024; Feng al., 2024), noise models for explicit likelihood inference (e.g.. Legin al., 2023; Adam et al., 2023), and density estimators in conjunction with deep emulators for simulation- based inference (e.g.. Cranmer et al., 2020: Rezende & Mohamed, 2015; Papamakarios al., 2021; Price et al., 2018; Greenberg et al., 2019; Papamakarios et al., 2019). However, the adoption of
*Corresonndene: Pablo. temos@sandbooxuuantumm com and sammy sha Shharief@nn effummntteee ca.

---

Published as a conference paper at ICLR 2025
Figure I: Illustration of the PQMass statistic. Left: Two sets of samples (xx) f p. and (yr); 4. shown blue and red points. For given region R X, the fraction of sam- ples in R for each set follows a binomial distribution. We can test the hypothesii that the two binomial distributions are the same. Right: Partitioning of the space into non-overlapping regions (R e.g. Voronoi tessellation, the distribution of samples in the regions follows multinomial distribution.
PQMass performs hypothesis tests on equality of the two multinomial distributions' parameters to obtain p-value.
these methods in scientific data analysis demands rigorous accuracy in uncertainty quantification. Specifically, it is crucial to verify that these generative models accurately represent the full distribution of their training data.
When evaluating generative models, one is interested in three qualitative properties (Stein et al., 2023; Jiralerspong et al., 2023): Fidelity refers to the quality and realism of outputs generated by model, i.e. how indistinguishable generated samples are from real data. Diversity pertains to the range and variety of outputs model can produce. For example, a model that misses a mode in the training data has lower diversity. Novelty refers to the ability of a model to generate new, previously unseen samples that are not replicas of the training data yet are coherent and meaningful within the context of the task. The trade-offs between these properties are complex and depend on the function class (e.g., model architecture) used for the generative model, the learning objective, properties of the data distribution, etc. (Stein et al., 2024). For example, samples from model that simply copies the training data will exhibit high fidelity and diversity but will lack novelty; on the other hand, an excessively smooth model will generate novel, diverse samples but will lack fidelity.
Two classes of methods for comparing generative models to ground truth distributions exist: sample- based methods, which compare generated samples to true samples, and seeiihodd-based methods, which make use of the likelihood of the data under the model. It has been observed that likelihood- based methods, such as negative log-likelihood (NI..) of test data, show too much variance to be useful in practice and are 'saturated' on standard benchmarks, iee., they do not correlate well with sample fidelity, particularly in high-dimensional settings Theis et al. (2016); Nalisnick et al. (2019); Nowozin et al. (2016); Yazici et al. (2020); Le Lan & Dinh (2021). On the other hand, most sample- based methods cannot measure the combination of fidelity, diversity, and novelty. For example, the Fréchet Inception Distance (FID; Heusel et al., 2017) and the Inception Score (IS; Salimans et al., 2016; Sajjadi et al., 2018) measure fidelity and diversity, but not novelty, while precision and recall (Salimans et al., 2018) () measure only fidelity and diversity, respectively; the authenticity score (Alaa et al., 2022) measures only novelty. Recently, the Feature Likelihood Divergence (FL.; Jiralerspong et al., 2023) was proposed
measure that captures fidelity, novelty, and diversity. However, FLD essentially relies on the approximation of the underlying distribution by a Gaussian mixture model and consequently requires feature extraction and compression in order to function in high-dimensional settings.
In this work, we propose PQMass, a statistical framework for evaluating the quality of generative models that measures the combination of fidelity, diversity, and novelty and scales well to moderately high dimensions, without the need of dimensionality reduction. PQMass allows us to accurately estimate the relative probability of a generative model given test dataset by examining the probability mass of the data over appropriately chosen regions. The main idea motivating PQMass is that given a measurable subset of the sample space, the number of data points lying in the subset follows a binomial distribution with a parameter equal to the region's 's (unknown) true probability mass. By extension, given partition of the space into regions, the counts of points falling into the regions follow a multinomial distribution. PQMass carefully defines partitions of the space into regions defined by Voronoi cells, quantizes sets of samples from two distributions into these cells, and uses two-sample tests on parameters of the resulting multinomial distributions. This approach provides a p-value that estimates the quality of generative model (the probability of a model given the training data). It also allows the comparison of various competing models by analyzing the ratio of their probabilities.
PQMass satisfies the following desiderata:
(1) Since PQMass works with the integrals of the true and estimated probability density functions over regions, it does not make any assumptions about the true underlying density field, e.g., its smoothness or approximability by simple families.
2

---

Published as a conference paper at ICLR 2025
(2) It is statistically principled, allowing to directly soour the answer the question: "What is the probability that two sets of samples were drawn from the same distribution?"
(3) PQMass is computationally efficient and scales well with the dimensionality of the data, allowing us to directly evaluate generative models of high-dimensional data, such as images. However, PQMass can also readily work in feature space if desired.
(4) PQMass is general and flexible, allowing the evaluation of generative models on any type of data where a metric can be defined, including images, tabular data, and time series.
2 THEORY AND METHODS
Our problem statement the following: We have two collections of i.i.d. samples (x:) . ~ p and (1)) Y; ~ q, where both p and q are probability distributions' on Rd. We are interested in testing the statistical hypothesis that p = 4, that is, the two sets of samples came from the same distribution.
To achieve this, we will develop tests on well-understood distributions derived from p and q that would upper-bound the probability that p = q and show that they almost surely distinguish two nonequal distributions as the number of samples increases.
2.1 TESTING EQUALITY OF QUANTIZED DISTRIBUTIONS USING S SAMPLES
Equality of distributions from equality of probabilities of regions. We begin with some elementary facts from probability and real analysis. Recall that two probability measures and y are equal they assign the same mass to all measurable sets, i.e.
(R) = (R)
VR V R measurable.
Because p and are Borel measures, it is equivalent to verify (1) only for open sets R, or a smaller generating set, such as all polyhedra or products of intervals (see, e-g., Rudin, 1987).
Let Rd R1 L R₂ LI ... L Rag be a partition of Rd into disjoint measurable regions. Define as the function that assigns to each point x E Rd the index of the region it belongs to, i.e., x E Rx(x)- If p is distribution on Rd then the pushforward measure x.p is distribution on (1.2...., nR), with mass function x, pj) = p RR)) This distribution is a quantized version of p. where samples
π Rdd -
replaced by the index of the region into which they fall.
With this definition, (1) can be restated: p= q if and only if x.p n. for all partitions Rd = R R2, or, more generally, Rd Ri u... U Rng (note that implicitly depends on the partition). As we will return to in $2.2, it is sometimes sufficient to consider restricted classes of partitions.
The main idea behind PQMass is that the probability that p = q is upper-bounded by the probability that p = F.G for given partition. Next, we will consider how to test the hypothesis that x.P π, using samples from p and q given a fixed partition. In $2.2 we discuss the choice of partition and propose an effective method.
Estimating probability masses of regions by sampling. Fix a partition Rd = =R L R2 U.... - Rux and define æ as above. If ~ p is random variable with distribution p. then л(x) is categorical variable with distribution x.p, i.e., taking the value j {1, 2,, .,,,......pp ,,R) with probability p p(R)).
Suppose now that Xj...... Xm p are i.i.d. samples from p. We then have (x;) ~ x.P for all i. Defining& & ((x)) R) = > 1 [x; E R], where the indicator function, as the number of samples in a region R, we have that ((x Rj) is a Binomial(m, m. p(R))) random variable. In particular, the proportion of samples in RJ is unbiased and strongly consistent estimator of p(R,), since the mean of k ((x))),, R,) /m is p(2)) and the law of large numbers implies that this estimate converges to p p(R)) almost surely as n - c0.
Generalizing, we have that the vector of counts in all the regions follows multinomial distribution: Rnp )))
k ((x))) == (k ((x)),, ......,, & ((x)
Rmn)) ~ Multinomial (m,
(2)
and the vector of empirical proportions of samples in the regions is an unbiased and strongly consistent estimator of the vector of true probabilities (p (R)), .,....., (Rng))
We assume p and q Borel measures, i.e., all open sets measurable. This is true if they are absolutely continuous with respect Lebesgue measure or the induced measure on embedded submanifold.
3

---

Published as a conference paper at ICLR 2025
Testing the equality of multinomial distributions. Suppose that (x)) p and (yr) Yr are i.i.d. samples from and respectively. According to (2), the vectors of counts k ((x)) i=1 and k (())))) follow multinomial distributions with parameters (m, (p(R)).... ..... ((Rng) mne) and (n, (g (1))....... . ((nn))) respectively. Testing the statistical hypothesis that x.P TT is the same as testing the hypothesis that the vectors (p(R))..... p Rmz)) and (q(R))..... (Rng)) are equal. Thus the problem is reduced to lesting the equality of two multinomial distributions given samples from each, which is standard problem in statistics. We discuss how to approach this problem from frequentist perspective.
In frequentist approach, there are multiple ways to measure whether two multinomial distributions are the same (see, e.g., Anderson et al., 1974; Zelterman, 1987; Plunkett & Park, 2019; Bastian et al., 2024). In this paper, we use the Pearson x" test (Rao, 1948; 2002).
We start by defining the probability that point randomly chosen from among the m + points (x;) (1)) falls region Ry:
Pj = k ((x)) R2) ((y)) Rj)
+
If the n points were randomly partitioned into two subsets of sizes and n, then the expected number of points in Rj within the two subsets would be
mpj.
N/2 np
respectively. This motivates the definition of the XPOM statistic using the definition of the Pearson cqi-squared:
k ((x)) R3) (D)) (a ((ri) R2) - )2)))) 2
nR
2 XPQ
(4)
N
is known that, given any distributions, this statistic follows x² distribution with ng degrees of freedom. Therefore, we can calculate the p-value of the test as
p-value
x (2) dz.
(5)
X
This p-value provides an estimate of the probability that the x.p = R. i.e., that the distributions quantized into the regions R coincide.
2.2 CHOOSING AN EFFECTIVE PARTITION
The test described above depends on the choice of partition, with some choices of regions giving more powerful test than others. For example, (5) is unlikely give meaningful p-values if nearly all the mass of both and is captured by one of the regions. Similarly, one could be 'unlucky' in the choice of regions and obtain that x.P happens to be very close to π.q, making the test uninformative with small number of points. Thus, one key element in PQMass is the choice of the regions Ry.
Voronoi cells. Inspired by the Tests of Accuracy with Random Points (TARP: Lemos et al., 2023a) framework for estimating posterior coverage, we propose to define the regions as Voronoi cells.
Recall that given a set of reference points 21....: Zux- the Voronoi cell corresponding to 2j is defined as the set of points that is closer to 2; than to any other zp. To be precise, we can break ties according to the index, yielding the following definition:
X E R2 - Vj', |||x - z < ||x - xxyll or [||x - z, = ||x - xy and j < j']]-
(6)
The sets Ri..... Rng then partition &d into nonoverlapping regions, and the function æ that maps points to the index of the containing region is defined by xxx) arg min, ||x zyll, breaking ties in favour of the lower index (see Fig. 1, right).
While the definition (6) uses the L distance metric, D Rd x Rd R. In Table we compare this choice to that of using the L metric, as well as with the choice of using metric defined in feature space.
define Voronoi cells using any metric
4

---

Published as a conference paper at ICLR 2025
Choice of centres. It remains specify how to choose the reference points 2j. To ensure that the regions approximately uniformly cover both distributions, we choose to sample the reference points from uniform mixture of the empirical distributions of samples from p and q.
For the validity of the test in $2.1, the choice of the regions should be independent of the points used to perform the test. Thus one can use small subset of the available sample points as reference points to construct a tessellation, then evaluate the XPOM statistic using the remaining points.
Algorithm. The full algorithmic instantiation of PQMass, given sets of points (x))," and (yr), choice of the number of regions nR I, and distance metric D, can be summarized as follows:
1. Define reference points: Set 2I...., Zng to a random sample from [ [ points from (x)) and [ng] points from (y)) Remove the choice points from the collections (xx) and (y²).
Count points in Voronoi cells: Obtain the vectors of counts k ((x:) and k ((v))
with respect to the regions defined by reference points Zj. by computing bincounts of arr min, D(x;, and arrg D(yi. ) respectively.
3. Test to compare multinomials: Compute the XToM and p-value via (4) and (5), respectively.
In practice, to obtain useful metric, can perform the test multiple times, with different sets of reference points, to reduce the variance resulting from the choice of regions.
2.3 CONSISTENCY GUARANTEES: WHAT INFORMATION IS LOST BY PQMASS?
We present results showing that the PQMass test is capable of distinguishing distinct distributions.
The first proposition shows that even with two regions, PQMass has nonzero probability of distin- guishing two distributions.
Proposition 2.1. Suppose that p. q. r are + that p and q have smooth densities, and that r has full support. Then, for nR = and references points Z1, 22 r, the probability that n.p π. is strictly positive and hence the PQMass test is consistent as m, n - oo with m/ n being bounded both above and below by strictly positive, finite scalars.
continuous distributions on Rd with
Proof. As m,n + co, the bounds on m/n guarantee that m and n grow at rates such that Fisher's test for multinomial distributions with m and samples from the two distributions distinguishes them. Because r has full support, the distribution of the hyperplane separating the two Voronoi cells defined with respect to samples from has positive density in the space of hyperplanes. Therefore, if the test statistic were almost surely zero, then the measures of almost all half-spaces under p and would coincide. By disintegration, this would imply that the marginals of and q coincide on almost all hyperplanes, ie., the Radon transforms of and are equal. This would imply the equality of p and q, contradiction.
Since π depends on r, the probability referred to in Prop. 2.1 is over the choice of ng samples from r. Of course, for "bad" choices of reference points, may be unlucky and not distinguish and q (if the two resulting cells have the same mass under p and q). The proposition states that with positive probability, this does not happen. Similarly, as the number of references points nR grows to infinity, the probability that the test distinguishes two distributions approaches one. This can be understood as a consequence of the fact that the Voronoi cells 'tessellate' the space more finely as nr grows.
Proposition 2.2. Let p. p.q.r be as in the previous proposition. As nr oo, the probability that g.P # n.q with respect to the choice of reference points 2I....,Znn ~ approaches I.
Znn
Proof. We abuse notation and use p and interchangeably with their densities. Select any xo such that pxx)) < q(xo), which exists because # and both p and q integrate to over Rd. By continuity, there exists 8 0 such that if ||x xoll 6, then p(x) qxx).
Let be the open ball of radius d about xo- By Theorem 3.1 of Gibbs Chen (2020), the probability that the diameter of the Voronoi cell (defined using reference points sampled i..d. from the full- support distribution r) containing Xp is less than approaches I as π - co. Any polytope of diameter less than 5 and containing Xg contained in B; thus, the probability that at least one region R2 is
5

---

Published as a conference paper at ICLR 2025
Null test. We generate sets of samples from the same Gaussian mixture model in 100 dimen- sions. We then measure XFOM value of our test
Figure 2:
125
(ng= 100) and repeat this process 2 times, resam- pling every time. We can see that the XPOM value is distributed as a chi-squared distribution with RR - degrees of freedom, as expected.
0.01
2045
4096
8192
9.02
2199)
8.01
60
80
160
120
140
160
x.
FOM
Table I: Comparison of various sampling methods on a mixture model in D = 2 and funnel in D = 10.
The results for PQMass are compared to the Wasserstein distance, the MMD with RBF kernel, and the Jensen-Shannon divergence.
Mixture Model (D 2)
Funnel (D = 10)
Model
XPOM (ng= 100)
W
RBF MMD
JSD
POM (ng 100)
W
RBF MMD
MCMC
1621.06 # 82.40
8.94
0.0151
0.42
855.40 49.60
17.41
0.0566
FAB
678.22 + 88.43
5.77
0.0038
0.39
96.69 + 13.52
22.08
0.0032
GGNS
109.32 + 12.36
3.20
0.0011
0.35
472.99 18.00
32.20
0.0358
contained in B approaches as - oo, Such a cell would have higher mass under than under p. which implies the result.
Note that the nR to compare two x² distributions with infinite degrees of freedom. The proposition states that the probability that the tessellation defined using ne random reference points sampled from separates and q which is a well-defined quantity for any "R approaches as AR - co. These results show that, least for 'well-behaved' densities, no information is lost by the PQMass statistic. By this, we mean that if two distributions are distinct, then the test has a positive probability of distinguishing them (and the probability approaches in the asymptotic limit). This contrasts with tests that approximate the distributions by certain fixed class (e.g., tests considering only summary statisties such as the recond-moment statisties used by FID). These results can likely be generalized further, which we leave for future work.
limit in Prop. 2.2 does not refer performing the test with "næ- co"
Since Prop. 2.2 is about the sensitivity of the test the asymptotic limit, the proposition simply tells us that the sensitivity of the test approaches us about the rate of convergence. The usefulness of the test itself is demonstrated by the empirical evidence in the experiments presented in the next section.
the number of samples grows, but does not tell
3 EXPERIMENTS
3.1 NULL TEST
We start by validating PQMass on null test, by comparing two sets of samples that are known to be equivalent. As shown in Fig. 2, we use a Gaussian mixture model in 100 dimensions with 20 components as our generative model. We then repeat the following process 2 times: We generate number of samples (see legend) from the Gaussian mixture model and then measure the XPOM value
of our test, with nı= = 100. The XrOM value is approximately distributed as a chi-squared distribution with ля - degrees of freedom, with the approximation getting better as the number of samples gets larger, expected from $2.1.
This test, while simple, is important, as it shows that the proposed method not biased towards rejecting the null hypothesis. We show further null tests for complex distributions in $A, and explore approximations that can be used when a limited number of samples are available in SI.
3.2 VALIDATION OF SAMPLING METHODS
In this section, we show how PQMass can be used to study the performance of sampling algorithms. Other sample-basee metrics commonly used to evaluate sampling methods include the Wasserstein distance and maximum mean diserepancy (MMD, Gretton al., 2012). While these sample-based

---

Published as a conference paper at ICLR 2025
Figure 3: Various sample-hased metrics as a function of the dimensionality of the data. We use mixture of 10 equally weighted Gaussians with varying numbers of dimensions. For each, we calculate the value of both metrics when comparing samples from the same dis- tribution (blue) and from different distributions where one of them is missing the test various times. Sample-base based metries should detect that a mode has been dropped.
3100
2000
mode (orange), repeating
Maustg aoak
I0PP XD-D
6022
[aver
-02
Figure 4: Computational cost of various sample- based metrics data dimensionality. We show the time required calculate the value of each metric the experiment Fig. 3. both as a function of the dimensionality of the data (left) and the number of samples (right).
Di MMDD 200 LOCD
Number of somplox
metrics are well-adapted to low- diweesnniional settings, they do not scale well to high-dimensional settings, as they may be sensitive to outliers or to the choice of kernels.
PQMass is consistent with baseline metries on simple tasks. First, we compare PQMass to other sample-based metrics to perform well. We use the two-dimensional Gaussian mixture model used in Midgley et al. (2023), which has been used for benchmarking various sampling methods. We generate samples using three representative methods: flow-annealed importance sampling bootstrap (FAB, Midgley et al., 2023) a Markov chain Monte Carlo (Metropolis et al., 1953) algorithm as implemented in emcee (Foreman- Mackey et al., 2013)?, and gradient- gradient-guided guided nested sampling (GGNS, Skilling, 2006; Lemos et al., 2023b). For each method, we generate 10, 000 samples and compare them with 10, 000 samples from the true distribution. We calculate the 2- Wasserstein distance, the mean maximum discrepancy with a radial basis function (RBF) kernel, and the kernel density estimation); and compare them to the XPOM value with пą 100. For PQMass, we repeat this process 20 times, resampling the reference points, and report the standard deviation. We show the results in Table PQMass is in good agreement, and the XPOM values correlate well with the other sample-based metrics, with further comparisons in SH.
oo-dimensional task where traditional sample-based metrics are known
divergence (after performing a step of
Scaling to more complex distributions. We then repeat this experiment on nigher-dimensions task: Neal's 10-dimensional funnel distribution (Neal, 2003), another common sampling benchmark. PQMass is again in good agreement with MMD, but not with the Wasserstein distance. We show samples from each model in &C, which show that, visually, PQMass correctly identifies the best- performing sampling methods.
PQMass detects mode-dropping in high dimension. Finally, we study the scaling of PQMass with the dimensionality of the data. We use a uniform mixture of 10 Gaussians in R for varying d. For each d, we generate 5, 000 samples and compare them with 5, 0.00 samples from the same distribution. We calculate the X value of our test, with ng 100, as well as other sample- based metrics. We then repeat the experiment, but comparing the true distribution with the distribution with one mode dropped. We show the results in Fig. 3. While W/ and linear MMD get increasingly noisier as the dimensionality increases, PQMass remains stable. Furthermore, the computational cost of radial basis MMD scales far worse with dimensionality than PQMass (Fig. 4). MMD with linear
Note that MCMC does not produce independent samples, as samples within the samples are correlated. While we randomly subsample 0.01 of the total number of samples in each chain, there could still be spurious correlations between these, affecting, the MCMC results. However, we find the MCMC results to be significantly worse than competing methods in these experiments, far heyond the possible effects of spurius correlations.
7

---

Published as a conference paper at ICLR 2025
Figure 5: Tracking training progress of generative mod- els on MNIST. After each training epoch, we generate samples and compute the XTOM value to assess their quality against test samples. We perform this experi- ment twice for each model, and see that the values agree even though unique samples were generated The diffusion model can achieve values closer to ideal OPQM equal to the number of regions "R 1, here 99). XFQM was evaluated by retessellating and resampling the data sets.
DAB Mlsdel Hus
10a
VAE Medell Bun
Putrostin Madel
Dutfuston Model
*k 10x
each run.
10a
40
Epoch
Figure 6: Correlation of human error rate in identifying real CIFAR-10 and FFHQ images with various sample- based metrics (results of prior methods taken from Ji- ralerspong et al. (2023)). Each point corresponds to single model. The result for PQMass is the XPOM value averaged over 20 samples. PQMass estimates the fidelity and diversity of generative models well despite not re- lying feature representation, unlike FL.D and FID.
chaan Heme
122
3.1
142
105
0.2
4.3
100
0.2 Dhcaus
2.2
0.3
Zhamsa
kernel the cheapest computationally, but, as shown in Fig. 3, it is too noisy to be usable even in simple high-dimensional problems. We show results with larger number of modes dropped in $F.II
3.3 ASSESSING IMAGE GENERATIVE MODELS
Tracking training progress on small images. In this section, we track the value XPOM between model-generated and ground truth samples as we train two generative models, a variational autoen- coder (VAE; Kingma Welling, 2014) and denoising diffusion model (Ho et al., 2020; Song et al., 2021) (see $D for model details). We train on the MNIST train set and compare generated samples with the MNIST test set. After each epoch of training, we generate 2000 samples and compare them with 2000 test samples by computing XPOM with пą 100. We execute this experiment over the first 100 training epochs to highlight the early stages of training for the models and repeat the experiment twice. As shown in Fig. 5, as training progresses, the metric stabilizes, with fluctuations due the stochasticity of training and the unique samples generated at each epoch.
As shown in 1, the VAEs fail to capture correct structures of the images, while the diffusion- generated images are more realistic. This is reflected in the low XFOM values for the diffusion model, which plateau at around X closer to = 99, the idcal value.
We remark that this test was performed directly in pixel space, without dimensionality reduction or feature extraction. Thus, at least for simple images, PQMass serve as userful metric for tracking the quality of samples from a generative model over the course training.
Measuring mode coverage. To verify that method can measure the diversity of samples from generative models, we retrain the variational autoencoder and diffusion model described above on subsets of MNIST where all images of N randomly chosen classes missing, for varying N. The results, averaged over random runs and samples of classes, are shown in Fig. 7. As expected, the value of XPOM increases as we drop more classes for both models. In addition, verify the ability of PQMass to detect the presence of small mode in one of the sets $F.2.
PQMass correlates with human judgments. Next, we consider the assessment of pretrained models on larger image datasets. We perform an experiment inspired by Stein et al. (2023); Jiralerspong et al. (2023), focusing on the CIFAR- 10 and FFHQ256 (Karras et al., 2019) datasets. Following Stein et al. (2023), we first use the human error rate as a measure of the fidelity of the generative models.
3 lecun Phtt://yam...cuu..come/d/mmmmii wttps///c.cooeeedd/////s////////
4 hhtps://www.cs wwrontoo wwwp://ee/////eeaaaaa.aaoaa hwt.::/ww...o.ooooteeeiriiiiiia.
8

---

Published as a conference paper at ICLR 2025
Figure 7: We show the results for PQMass validating generative models trained on MNIST on the Y-axis, and the number of dropped modes on the X-axis, for a varia- tional autoencoder blue, and a diffusion model red. The xPoM value increases as we drop more modes, as expected. For this experiment, XPOM was evaluated by retessellating and resampling the data sets.
30
oa Hight
Table 2:
We compare ImageNet train data, noised with Gaussian noise of different variances, to ImageNet test data. Wc also ran FLD for the same samples using the Dino feature extractor. For PQMass, these experiments were done by retessellating and resampling the for evaluating XPQM As compared to FLD, PQMass is more
sensitive the noise despite not using feature extractor.
Nakee o²
0.50
0,01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.10
0.15
0.20
0.50
1.00
95.55
107.63
116.63
131.55
151.67
212.25
227.44
265.38
352.49
766.00
1252.45
2818.56
4455.07
9.29
9.15
9.27
10.54
10387
10.96
15.84
21.79
42.78
We compare generated samples from each of the models the test data for both datasets. We repeat the comparison 20 times, each time varying the reference points. We show the results in Fig. 6. We find that our chi-squared values visibly correlate with the human error rate, indicating that PQMass can effectively measure the fidelity and diversity of generative models.
We note again that these experiments were performed pixel space. One key strength of PQMass is its scaling to high-dimensional problems: methods such as FID and FLD fail in high dimensions without the use a feature extractor to reduce the dimensionality of the data, while POMass continues to give meaningful results, at much lower computational cost and without introducing potential biases through feature extraction. This ability is important because pretrained feature extractors do not exist for many modalities of data that appear in important scientific applications.
However, PQMass can also work in feature space: Table 3 shows a comparison of PQMass applied in pixel space and using Inception V3 features (Szegedy al., 2016).
PQMass effectively detects small noise. Next, we consider the detection of additive noise on ImageNet (Deng et al., 2009). We first PQMass on subset of the training data with added Gaussian noise of different variance, comparing it to the test data (see $E.2 for visualization). We repeat the experiment by replacing PQMass with FLD using the D2 (Oquab et al., 2024) feature extractor. The results of both experiments are shown in Table 2. PQMass is more sensitive to noise, especially in the high noise regime, demonstrating that PQMass not only scales well with dimension (see $F.5 for further demonstrations of the scalability of PQMass to high- -igh-dimensional settings), but also provides a measure of distribution shift caused by increasing noise added to the data.
Using PQMass to measure novelty and memorization. While PQMass can measure the fidelity and diversity of moodel-generated samples, it does not directly measure their novelty. One approach for detecting memorization the generalization gap for PQMass, i.e., how this metric differs between comparing the generated samples to the training data and comparing the generated samples to the validation data. However, propose a way to detect overfitting from XPOM on the training data directly. Indeed, we know that a value that is too low is indicative of overfitting. For large number of regions nR (i.e., nR degrees of freedom), we know the chi-square distribution is approximately Gaussian (Fig. 2). Therefore, can use the asymunetry of this distribution to get a p-value that penalizes suspiciously low values of X by considering the mirror reflection of the chi-square value around the maximum:
overfitting would be to follow Jiralerspong et al. (2023) in looking at
2nR X
p-value (XPOM)
To study the effectiveness of this metric, we repeat the 'copycat' experiment of (Jiralerspong et al., 2023): We generate samples using one of the well-performing CIFAR-10 models used above (PFGMPP, Xu et al., 2023) and repeat our test, replacing varying fractions of the generated samples with samples from the training data. We then calculate XPOM first comparing the generated samples to the training data and then the validation data.

---

Published as a conference paper at ICLR 2025
Figure 8: Novelty. PQMass (Y-axis) when fraction of the generated samples replaced by samples taken from the training set (X-axi)). We repeat the compari- son 20 times, each time changing the reference points, and report the standard deviation. We show the value of XTOM in the top left panel, for the test (red) and train (bluc), and the p-value for the training data on the top right. We see that, as memorization increases, the gap between the train and test XPQ increases, and the p-value goes down. The bottom pannels compare peerl (X (PMM) with the Cr score and the per- centage authentic samples authpet, two metrics thats measure novelty.
178-
Tot
Toun
100
044
0.5
0.5
09
x
X
We show the results for in Fig. 8. The top left panel shows that the gap between XPOM when comparing to the training data and when comparing to the validation data increases with the number of samples we replace. The top right panel shows the two-sided p-value, which decreases as the number of copied samples increases. These results show that both generalization gap and two-sided p-value can be used to detect memorization in generative models. The bottom two panels show the effectiveness of the p-value in (7) in capturing memorization and its agreement with an established metric, the authenticity score authpct. (Alaa et al., 2022).
Experiments the Appendix demonstrate the effectiveness of PQMass for other data modalities, including dimens-dimensional images from scientific applications ($F.5), time series ($F.3), tabular data ($F.4) and protein sequences ($F.6).
LIMITATIONS
While PQMass performs well on various tasks, is important to consider its limitations.
PQMass was shown to perform well with large number of samples. The statistics, however, will be too noisy for practical use if this number is too limited. Similarly, the number of regions is important: while our ablation study ($B) shows that the results are robust to various choices, in the limits ng = and ng + N (where N is the number of samples), sensitivity will be limited.
Second, PQMass could fail for fixed choice of reference points defining a tessellation that fails to discriminate two distributions well. For this reason, repeating the experiment for various randomly chosen tessellations is recommended, as this effectively marginalizes the choice of tessellation (as have done in this work). Fortunately, this is feasible, as the computational cost of PQMass is low.
Third, PQMass does require i.i.d. samples. Therefore, if generative model produces correlated samples, the PQMass formalism cannot be used.
Finally, POMass is only good as the distance metric used, as shown in SG. For some modalities (for example, natural language), it is nontrivial to select a meaningful distance metric between samples.
CONCLUSION
In this paper, we have introduced a new method for quantifying the probability that two sets of samples are drawn from the same probability distribution. PQMass is based on comparing the probability mass of the two sets of samples in a set of non-overlapping regions. It relies only on the calculation of distances between points and, therefore, can be applied to high-dimensional problems. It does not require training or fitting any models. Furthermore, it does not require any assumptions about the underlying distribution and, therefore, can be applied to any type of data as long as one has access to true samples. We have shown that PQMass can be used to evaluate sampling methods and track the performance of generative models as they train.
We have shown the performance of PQMass sampling methods, comparing generative models of images, and detecting hidden signals in time series. Given the versatility (further shown in the additional experiments in $F) and low computational cost of PQMass, can serve as a valuable tool for evaluating the quality and performance generative models and sampling methods.
variety of synthetic tasks, as well comparing
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was made possible by generous donation by Eric and Wendy Schmidt with the recommendation of the Schmidt Futures Foundation. This work is also supported by the Simons Collaboration "Learning the Universe". The work is in part supported by computational resources provided by Calcul Quebec and the Digital Research Alliance of Canada. Y.H. and L.P. acknowledge support from the Canada Research Chairs Program, the National Sciences and Engineering Council of Canada through grants RGPIN-2020- 05073 and 05 1010 and the Fonds de recherche du Québec through grants 2022-NC-301305 and 300397. N.M. acknowledges funding from CIFAR, Genentech, Samsung, and IBM.
We thank Adam Coogan, Sebastian Bigner-Carenal Joey Bose, Gauthier Gidel, Alex Tong, Alexandra Volokhova, and Marco Jiralespong for very inspiring discussions and feedback on early versions of this work. We also thank Marco Jiralespong for sharing useful data.
REFERENCES
Alexandre Adam, Connor Stone, Connor Bottrell, Ronan Legin, Yashar Hezaveh, and Laurence Berreault-Levasseur Echoes in the Noise: Posterior Samples of Faint Galaxy Surface Brightness Profiles with Score-Based Likelihoods and Priors. In Machine Learning and the Physical Sciences Workshop, NeurIPS 2023, 11 2023.
Suzanne Aigrain and Daniel Foreman foreman-Mackey. Gaussian process regression for astronomical time series. Annual Review of Astronomy and Astrophysics, 61:329-371, 2023.
Ahmed M. Alaa, Boris van Breugel, Evgeny Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. International Conference on Machine Learning (ICML), 2022.
Loic Albert, David Lafrenière, Doyon René, Étienne Artigau, Kevin Volk, Paul Goudfrooij. André R. Martel, Michael Radica, Jason Rowe, Néstor Espinoza, Arpita Roy, Joseph C. Filippazzo, Antoine Darveau-Bernier, Geert Jan Talens, Anand Sivaramakrishnan, Chris J. Willott, Alexander W. Fuller- ton, Stephanie LaMassa, John B. Hutchings, Neil Rowlands, M. Begoña Vila. Julia Zhou, David Aldridge, Michael Maszkiewicz, Mathilde Beaulieu, Neil J. Cook, Caroline Piaulet, Pierre-Alexis Roy, Pierrot Lamontagne, Kim Morel, William Frost, Salma Salhi, Louis-Philippe Coulombe, Björn Benneke, Ryan J. MacDonald, Doug Johnstone, Jake D. Turner, Marylou Foumier-TTondreau. Romain Allart, and Lisa Kaltenegger. The Near Infrared Imager and Slitless Spectrograph for the James Webb Space Telescope. III. Single Object Slitless Spectroscopy. Publ. Astron. Soc. Pac., 135(1049):07500, July 2023.
Donald A Anderson, Lyman McDonald, and Kim Weaver. Tests on categorical data from the unionintersection principle. Annals of the Institute of Statistical Mathematics, 26:203-211 1974.
Patrick Bastian, Holger Dette, and Lukas Koletzko. Testing equivalence of multinomial distribu- tions constrained bootstrap approach. Statistics & Probability Letters, 206: 109999, 2024.
Connor Bottrell, Hassen M. Yesuf, Gergö Popping, Kiyoaki Christopher Omori, Shenli Tang, Xuheng Ding, Annalisa Pillepich, Dylan Nelson, Lukas Eisert, Hua Gao, Andy D. Goulding, Boris S. Kalita, Wentao Luo, Jenny E. Greene, Jingjing Shi, and John D. Silverman. IllustisTNNG in the HSC-SSP: image data release and the major role of mini mergers drivers of asymmetry and star formation. Monthly Notices of the Royal Astronomical Society, 527(3):6506-655) 6639, January 2024. doi: 10. 10.19///naa////dd297111.
Ngai Hang Chan. Time series: applications to finance. John Wiley Sons, 2004.
Hyungjin Chung and Jong Chul Score-based diffusion models for accelerated MRI. Medi- cal Image Analysis, 80: 10:47747, 2022. ISSN 1361-8415. doi: https://doi.or//10..r 1016//..0.01.1mmm.. 2022.102479. URL https:/ www.scieecedirec.. wccm///ceennee/ttiiieee//e/ $1361841522001288
Hyungjin Chung. Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. International Conference on Learning Representations (ICLR), 2023.
11

---

Published as a conference paper at ICLR 2025
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-bart inference. Proceedings of the National Academy of S'iiences, 117(48):30055-3006 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. Computer Vision and Pattern Recognition (CVPR), 2009.
Noe Dia, M. J. Yantovski-Barth Alexandre Adam, Micah Bowles, Pablo Lemos, Anna M. M. Scaife, Yashar Hezaveh, and Laurence Perreault-Levassee Bayesian Imaging for Radio Interferometry with Goore-Based Priors. In Machine Learning and the Physical Sciences Workshop, NeurIPS 2023, 11 2023.
Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. International Conference on Learning Representations (ICLR), 2024.
M. Drozdova, Kinakh, O. Bait, O. Taran, E. Lastufka, M. Dessauges-Zavadsky. T. Holotyak, D. Schaerer, and S. Voloshynovskiy. Radio-astronomical image reconstruction with conditional denoising diffusion model. A&A, 683:A105, March 2024. doi: 10.1011/0004-336612222274994
Berthy T. Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L. Bouman, and William Freeman. Score-based diffusion models as principled priors for inverse imaging. International Conference Computer Vision (ICCV), 2023.
Berthy T. Feng, Katherine L. Bouman, and William T. Freeman. Event-horiizon-saal imaging of M87* under different assumptions via deep generative image priors. arXiv preprint arXiv:2406.0222 02785, 2024.
Thomas Flöss, William R. Coulton, Adriaan J. Duivenvoorden, Francisco Villaescusa-Navare and Benjamin D. Wandelt. Denoising Diffusion Delensing Delight: Reconstructing the Non- Gaussian CMB Lensing Potential with Diffusion Models. arXiv preprint vrXiv:2405.05598 2024.
Daniel Foreman-Mackey, David W Hogg, Dustin Lang, and Jonathan Goodman. emcee: the meme hammer. Publications of the Astronomical Society of the Pucific, 125(925):306, 2013.
Isaac Gibbs and Linan Chen, Asymptotic properties of random Voronoi cells with arbitrary underlying density. Advances in Applied Probability. 52(2):pp. 655-680 2020.
Phillip Good. Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer Science & Business Media, 2013.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Neural Information Processing Systems (NIPS), 2014.
David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood-free inference. International Conference on Machine Learning (ICML), 2019.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):7237773, 2012.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. Neural Information Processing Systems (NIPS), 2017.
Briland Hitaj, Giuseppe Ateniese, and Fernando Perez-Cruz. Deep models under the gan: information leakage from collaborative deep learning. Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp. (13, 2017. 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Neural Information Processing Systems (NeurIPS), 2020.
Marco Jiralerspong, Avishek Joey Bose, Ian Gemp, Chongli Qin, Yoram Bachrach, and Gauthier Gidel. Feature likelihood score: Evaluating generalization of generative models using samples. Neural Information P Processing Systems (NeurIPS), 2023.
12

---

Published as a conference paper at ICLR 2025
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Computer Vision and Pattern Recognition (CVPR), 2019.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022. URL
https: wwwppeeeeeeeeeeeeoe/ee/ee net/
Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. International Conference Learning Representations (ICLR), 2014.
Alistair Knott, Dino Pedreschi, Toshiya Jitsuzumi, Susan Leavy, David Eyers, Tapabrata Chakraborti, Andrew Trotman, Sundar Sundareswaran, Ricardo Baeza-Yates, Przemyslaw Biecek, et al. Al content detection the emerging information ecosystem: new obligations for media and tech companies. Ethics and Information Technology, 26(4):1-14, 2024.
Charline Le Lan and Laurent Dinh. Perfect density models cannot guarantee anomaly detection. Entmpy, 23(12):1690) 2021.
Ronan Legin, Alexandre Adam, Yashar Hezaveh, and Laurence Perreault-Levaaseurr. Beyond Gaussian Noise: A Generalized Approach Likelihood Analysis with Non- Gaussian Noise. The Astrophysical Journal Letters, 949(2):L41, June 2023. doi: 10.384772041188111cdd664.
Pablo Lemos, Adam Coogan, Yashar Hezaveh, and Laurence Perreault- Sampling baaseinn-bbsedd based accuracy testing of posterior estimators for general inference. International Conference on Machine Learning (ICML), 2023a.
Pablo Lemos, Nikolay Malkin, Will Handley, Yoshua Bengio, Yashar Hezaveh, and Laurence Improving gradient-guided nested sampling for posterior inference. Interna- tional Conference on Machine Learning (ICML), 2023b.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123-1133 2023.
Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087-1022, 1953.
Laurence Illing Midgley, Vincent Stimper, Gregor NC Simm, Bernhard Schölkopf, and José Miguel Hernández-Lobato. Flow annealed importance sampling bootstrap. International Conference Learning Representations (ICLR), 2023.
Gabriel Missael Barco, Alexandre Adam, Connor Stone, Yashar Hezaveh, and Laurence Perreault- Levasseur. Tackling the Problem of Distributional Shifts: Correcting Misspecified, High- Dimensional Data- Driven Priors for Inverse Problems. arXiv e-prints, art. arXiv:2407.17667, July 2024. doi: 00445500aaXxv.22407.7666)
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? International Conference on Learning Reprexentations (ICI.), 2019.
Radford M Neal. Slice sampling. The annals of statistics, 31(3):705-767, 2003.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural samplers
using variational divergence minimization. Neural Information Processing Systems (NIPS), 2016.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions Machine Learning Research, 2024.
George Papamakarios, David Sterratt, and lain Murray. Sequential neural likelihood: Fast likelihood- free inference with autoregressive flows. Artificial Intelligence and Statistics (AISTATS), 2019.
13

---

Published as a conference paper at ICLR 2025
George Papamakarios, Eric T Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Mach, Learn. Res., 22(57):1-64, 2021.
Amanda Plunkett and Junyong Park. Two-sample test for sparse high-dimensional multinomial distributions. Text, 28:804-826, 2019.
Leah Price, Christopher C Drovandi, Anthony Lee, and David Nott. Bayesian synthetic likelihood. Journal of Computational and Graphical Statistics, 27(1):1-11, 2018.
Radhakrishna Rao. Tests of significance in multivariate analysis. Biometrika, 35(1/2):58-7) 1948.
CR Rao. Karl pearson chi-square test the dawn statistical inference. Goodness-of-fi tests and model validity, pp. 9-24, 2002.
B. Remy, F. Lanusse, Jeffrey, J. Liu, J. Starek, K. Osato, and T. Schrabback. Probabilistic mass-mapping with neural score estimation. A&A, 672:A51, April 2023. doi: 10.1051/0004-6311/ 202243054.
Hansheng Ren, Bixiong Xu, Yujing Wang, Chao Yi, Congrui Huang, Xiaoyu Kou, Tony Xing, Mao Yang. Jie Tong, and Qi Zhang. Time-series anomaly detection service at Microsoft. Knowledge Discovery and Data Mining (KDD), 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. International Conference on Machine Learning (ICML), 2015.
François Rozet and Gilles Louppe. Score-based Data Assimilation. Neural Information Processing Systems (NeurIPS), 2023.
Walter Rudin. Real and complex analysis, 3rd ed. McGraw-Hill, Inc., USA, 1987. ISBN 0070542341.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Neural Information Processing Systems (NIPS), 2018.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Neural Information Processing Systems (NIPS), 2016.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving GANs using optimal transport. International Conference on Learning Representations (ICLR), 2018.
Mark F. Schilling. Multivariate two-sample tests based on nearest neighbors. Journal of the American Statistical Association, 81 81(395):799-800 1986. ISSN 01621459. URL http:// http::///ww./.oo.r.oo..ooo org/ atr/staalle/22800.
Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. Financial time series forecasting with deep learning: A systematic literature review: 2005-2019. Applied soft computing, 90:106181, 2020.
Kamran Shaukat, Talha Mahboob Alam, Suhuai Luo, Shakir Shabbir, Ibrahim A Hameed, Jiaming Li. Syed Konain Abbas, and Umair Javed. A review of time-series anomaly detection techniques: A step to future perspectives. In Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume I, pp. 865-877. Springer, 2021.
John Skilling. Nested sampling for general bayesian computation. 2006.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion digital forgery? investigating data replication in diffusion models. Computer Vision and Pattern Recognition (CVPR), 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations (ICLR), 2021.
14

---

Published as a conference paper at ICLR 2025
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. International Conference on Learning Representations (ICLR), 2022.
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, J Eric T Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Neural Information Processing Systems (NeurIPS). 2023.
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Neural Information Processing Systems (NeurIPS). 2024.
Connor Stone and Stéphane Courteau. The Intrinsic Scatter of the Radial Acceleration Relation. The Astrophysical Journal, 882(1):6, September 2019. doi: 10.3847/1338-435/bb111
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. Computer Vision and Pattern Recognition (CVPR), 2016.
Gábor Székely, Maria L Rizzo, et al. Testing for equal distributions in high dimension. InterStat, (16.10) (16.10):12 12491229-111 1272, 2004.
Lucas Theis, Aäron den Oord, and Matthias Bethge. A note the evaluation of generative models. International Conference on Learning Representations (ICLR), 2016.
Lei Xu, Maria Skoularidou, Alfredo Infaate and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. Neural Information Processing Systems (NeurIPS), 2019.
Yilun Xu, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, and Tommi Jaakkola. Pfgm++: Unlocking the potential of teoysics-inspire generative models. International Conference on Machine Learning (ICML), 2023.
Zhiwei Xue, Yuhang Li, Yash Patel, and Jeffrey Regier. Diffusion Models for Probabilistic Deconvo- lution of Galaxy Images. arXiv preprint arXiv:2307.11122, 2023.
Yasin Yazici, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, and Vijay Chandrasekhar, Empir- ical analysis of overfitting and mode drop in gan training. International Conference on Image Processing (ICIP), 2020.
Barak Zackay, Tejaswi Venumadhav, Javier Roulet, Liang Dai, and Matias Zaldarriaga. Detecting gravitational waves in data with non-stationary and con-gaussian noise. Physical Review D, 104 (6):063034, 2021.
Daniel Zelterman. Goodness-of-fin tests for large sparse multinomial distributions. Journal of the American Statistical Association, 624399)) 1987.
15

---

Published as a conference paper at ICLR 2025
MSIST
VAB
0.333
0.330
0.935
0.355
0.320
0.33
0.355
0.33
0,333
0.390 60
110
<igge
Figure 9: Null tests. similar to the one shown in Fig. 2, but for two different distributions: Samples from MNIST (left) and VAE trained on MNIST (right). The main result of our paper holds for samples from any distribution as long as we are comparing two sets of independent samples from the same distribution. Note that, on MNIST, the fit is not perfect due to the limited number of data points.
A NULL TESTS
The main result of this paper, shown in $2, was shown in practice in $3.1 for high-dimensional mixture of Gaussians. In this section, we show that works in even more complex and high- dimensional distributions. First, we test it on the MNIST data: Using the MNIST observations themselves as samples from some underlying probability distribution. We put together the MNIST train and test data, leading to 70, 000 images. We then split them into two subsets of 35, 000 images each. To repeat the null test of $3.1, we need independent samples at every iteration (note that this is different from the error bars reported in the rest of the experiments, which just arise from repeating with random tessellations, but always the same samples). Therefore, for each iteration, we take 1, 000 images from each dataset, and perform the PQM test with nR 100, leading to 35 data points. We plot their histogram in the left panel of Fig. 9. While noisy, due to the very few data points, we see that the histogram does follow a chi-squared distribution with nR degrees of freedom, as expected.
For our second experiment, we use one of the MNIST generative models described in $3.3. In this case, we can generate as many samples as we want. Because we are comparing the generative model to itself, the similarity of the generated samples to the MNIST train or test data is irrelevant for this particular test.
In practical applications, the number of samples available from real data might be limited, and in such cases, the PQMass distribution can be approximated by reusing the samples and only retessellating, which also yields good results. In practice, we have observed that, for typical sample sizes, the match between the X²oM distribution obtained by retessellating without resampling and the theoretical x² distribution is very close and more than adequate for testing the null hypothesis (we have indicated in the captions of the figures and the tables the specific experiments where this was done). However, in the cases where reusing the samples between retessellation, the retessellations are not independent since the samples shared between each test, which might affect the obtained distribution. We explore further the impact of this approximation in SI.
B ABLATION STUDY
In this section, we study the effect of varying the two hyperparameters of our experiment: The number of reference points ng. and the distance metric (which in all experiments in the main text is L2). We repeat the experiment on CIFAR10 described in $3.3, varying the number of reference points, as well as changing the distance metric LI. We also show the results when using the Inception-v3 feature extractor (Szegedy et al., 2016), as done in previous work such as Jiralerspong et al. (2023). We find that the order the models is robust to the choice of the number of reference points and the distance metric, well to the use of a feature extractor. Any observed differences are within the standard deviation of the experiment. We show the results in Table 3.
C SAMPLES FOR VALIDATION
In $3.2, we used PQMass to compare samples from various sampling algorithms to true distribution samples. Fig. 10 shows the samples from each algorithm and the underlying distribution. The
16

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study for CIFAR-10. We repeat the experiment of $3.3, varying the number of reference points, as well as changing distance metric to LI. show results for the frequentist version PQMass. sorted in order of decreasing XPOM for the default version. We find that the results robust to the choice of the number of reference points and the distance metric. Any observed differences are within the standard deviation of the experiment.
Model
Default (zı 100)
50
AR 200
LI Distance
InceptionV3
ACGAN-Mod
12407 + 10
10410 808
14125 = 995
12336 + 944
7891 = 1009
LOGAN
713 = 72
408 = 69
1180
728 + 108
5750 : 535
BigGAN-Deep
268 = 30
135 16
498 36
238
615 : 74
iDDPM-DDIM
238 = 22
143 17
470
21
533 = 78
MHGAN
234 39
123 19
470 48
194 18
628 64
StyleGAN-XL
230 = 13
119 15
477 23
203 = 23
199 + 19
StyleGAN2-ada
207 26
128 25
417 48
197 23
259 30
PFGMPP
177 = 22
108 = 18
335 = 28
175 + 16
239 + 28
MCMC xigu IR
FAD, xisa 678
GCNS, tigu 200.22
Truth
MCMC, xiqu 830
EAD, 1dgu 96,00
GCNS, thou 472.90
Truth
B
Figure 10: Samples from our the various sampling methods introduced in $3.2, and their corresponding PQMass values.
two distributions (gaussian mixture model and funnel) are often used for benchmarking sampling algorithms because of the high multimodality of the former and the complex shape of the latter. For Gaussian mixture model (top), we see that MCMC is missing mode, while the FAB samples are too noisy; for Neal's funnel (bottom), we see that FAB's samples look indistinguishable from true samples, while GGNS look similar but more spread out, while MCMC fails to model this distribution. In both cases, the results from XPOM correlate well with the similarity to true samples eye.
by
D GENERATIVE MODEL HYPERPARAMETERS
We detail the hyperparameters used for the image generative models trained in the main text. The MNIST VAE trained with Rel.U-activated MI.P encoder with layer structure (28 28)
512 - 256 - 20, and the decoder architecture was symmetric. The model was trained using the Adam optimizer with the learning rate initially For the denoising diffusion model, we used the package score score_modeell 1s Our model utilizes the NCSN+ architecture with variance- preenccii noising process and is trained with batch size 256 and learning rate 0.001 (with exponential moving average decay of 0.999. Neither model deviates from standard practices and we expect similar results to hold when assessing training progress of other generative models.
0.001, decaying by factor of 0.9 every 50 steps.
17

---

Published as a conference paper at ICLR 2025
E VISUAL INSPECTION OF SAMPLES
E. MNIST TRAINING PROGRESS
In the first part of $3.3, we trained four generative models, two (identical) VAEs, and (identical) diffusion models on the MNIST dataset for 100 epochs. In Fig. 5, we show the evolution of the XPOM values as training progresses. Here, in Fig. 1I, we display one sample from run of each generative model as function of the training epoch to provide qualitative understanding of the sensitivity of PQMass.
Autoenoider
5
6
1
7
6
5
3
7
2
9
Figure 11: Samples from the run diffusion model (left) as well as from the run VAE model (right) as function of training epoch.
E.2 ADDING NOISE TO IMAGENET
In $3.3, we evaluate PQMass's 's and FLD's ability to detect the addition of Gaussian noise with increasing variance compared the test set from the low to high noise regime. As shown Table PQMass is more sensitive to noise corruption than FLD. We display example showcasing the impact of each noise level on the sample in Fig. 12 to offer an intuitive understanding of the sensitivity of each metric.
Nate
Xate 001
Nate 602
Sate
Xite no0
Nate 003
Xote
Nate
Noae 008
Nowe 0.00
New 0.10
Nowe 0.15
Yvie 0:20
New 0.30
Nowr
Figure 12: A sample from the ImageNet dataset shown with Gaussian noise of increasing variance added, significantly altering the image in the high noise regime.
ADDITIONAL EXPERIMENTS
F.I GAUSSIAN MIXTURE MODEL
To evaluate our method's s ability to detect the diversity of generated samples, we reuse the Gaussian Mixture Model from $3.1, but this time we systematically remove samples from modes, for varying N. For each N, generate 5,000 samples and run test 20 times with nı 100, using different random reference points in each trial. The results are shown in Fig. 13. As expected, the XPoM value increases as more modes are dropped, demonstrating that PQMass can effectively detect when generative model fails to capture all modes, indicating reduced sample diversity.
18

---

Published as a conference paper at ICLR 2025
Gaussian Mixture Model. We show the results PQMass on axis, dropped modes on the X axis. We repeat the compari- son 20 times, cach time changing the reference points, report the standard deviation. The XPOM value increases as we drop more modes, as expected. this experiment, XPOM was evaluated by retessellating and regenerating new data sets.
Figure 13:
number of
60
20
Inogedt
F.2 DETECTING A SMALL MODE
We evaluate the effectiveness of PQMass at detecting discrepant small mode between two sets of samples. To this end, we use the CIFAR-10 dataset. We select eight classes (airplanes, automobiles, birds, cats, deer, dogs, frogs, and horses) and split the dataset into sets X and Y, each containing 24,000 samples (3,000 from each class). To simulate a small mode, we introduce parameter, a, and add 3000 images of new class (trucks) only X, while randomly removing the same number of images from other classes to keep the sizes X and identical. We then use PQMass as well as other existing tests compare X and
In Table 4, we see that, as we include images from the new class, which introduces a small mode, PQMass can detect the difference between X and Y even for small On the other hand, FLD struggles to detect the difference; FID also detects the difference (as shown by the monotonically increasing test statistic), but, unlike PQMass, does not give a point of reference allowing to compute significances (p-values).
We also should note that failing detect that two distributions differ by a small mode as described above could affect any two-sample test by construction. One can always design mode to be "sufficiently small" that it eludes a sample-based test, as samples from it would not appear among the test samples. It is also worth noting that in the Voronoi binning step of the PQMass test, all samples are binned, such that PQMass can be sensitive to a small missing mode even without reference sample within said mode.
Table Small mode detection. compare sensitivity of PQMass with FID and FLD at detecting a small mode using the CIFAR-10 10 dataset. The PQMass test was performed by retessellating and ressampling for every value of a.
PQMass
FID
FLD
PQMass
FID
FLD
0.00
100.41
17.66
-4.05
0.10
114.65
21.41
-3.88
0.01
100.60
17.77
-4.13
0.20
144.75
27.84
-3.78
0.02
101.71
18.03
-3.89
0.30
195.67
36.18
-3.85
0.03
102.71
18.31
-4.16
0.40
261.46
45.21
-3.98
0.04
103.80
18.68
-3.64
0.50
318.96
54.49
-3.95
0.05
104.02
18.96
-4.05
0.60
407.73
64.41
-3.69
0.06
106.89
19.56
-3.78
0.70
486.08
75.08
-3.22
0.07
108.00
20.15
-4.15
0.80
579.01
85.00
-3.37
0.08
110.44
20.40
-4.03
0.90
677.07
97.70
-3,01
0.09
112.31
21.11
-4.14
1.00
778.88
108.72
-2.90
F.3
TIME SERIES
For next experiment, we show the flexibility of PQMass by applying to different data modality: time series data. For this, we design an experiment where we observe a noisy time series of fixed length and aim determine whether underlying signal is hidden within the noise. The time series
19

---

Published as a conference paper at ICLR 2025
Figure 14: Time Series. We show the results for PQ- Mass on the Y-axis and the amplitude the signal on the X-axis. We repeat the comparison 5000 times, each time generating new time series with A = 0, and new time series with
We can see that the XPOM value increases grows, as expected. We also show the values of XPOM corresponding to the 30r and 5α significance levels of detection as black dashed lines.
Anseiaad
No signal
Signal
M MM MWM
14
0
2
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
Figure 15: Examples of time series with A = 0 (left), and with A = 0.12 (right). PQMass can detect the signal with 5α significance, signal that is invisible to the naked eye (at least, to those of the authors).
generated as follows:
y(1) = A cos(1) + 17(1).
(8)
where A is the amplitude of the signal and 17(1) is i.i.d. Gaussian noise with zero mean and unit variance. If = O, then the time series is just noise. If * O, then there signal hiding in the noise. For each observation, we generate 100 data points between r 0 and r = 10. We then repeat the following process 5000 times: We generate time series with A and time series with We then compare the two time series using PQMass, with næ = 100. We show the results in Fig. 14 for varying values of A. We can see that the XFOM value increases as A grows, as expected. The plot
also the values of XPOM corresponding the 30 and 5o significance levels of detection. We see that, for this experiment, we detect the signal with So significance for 0.12, signal that is invisible to the naked eyc. We show an example of a time series with this amplitude, compared to one without signal in Fig. 15.
This experiment demonstrates the versatility of PQMass. Because we make no assumptions about the underlying distribution, we can apply PQMass to any type of data as long as we have access to samples. Detecting signals in noisy time series is a common problem in multiple disciplines, such as astronomy (Zackay et al., 2021; Aigrain & Foreman- Mackey,, 2023), finance (Chan, 2004; Sezer et al., 2020), and anomaly detection (Ren et al., 2019; Shaukat et al., 2021). 1). Existing methods rely on assumptions about the underlying distribution. PQMass, on the other hand, can detect that the observed signal is not consistent with samples of random noise with no assumptions the generative process. We leave the application of PQMass to these domains for future work.
F.4 TABULAR DATA EXPERIMENT
To show the versatility of our method on different types of data, we applied PQMass to the generation of tabular data. We used CTGAN Xu et (2019) as our generative model, trained on data from the Adult Census Dataset 6, We convert categorical entries into one-hot encoding for distance metrics
hnttps//arcciis iss. usi /ddh//atasstt////aau
20

---

Published as a conference paper at ICLR 2025
14000
12000
XPQM
8000
H
6000
1
3
2000
60
70
Nuraber of epochat
Figure 16: XPOM for generative model tabular data, training progresses.
Prequeacy
0.073
0
9.6
0.223
0.00
15
20
25
50
0000
kdou 25
30
Figure 17: the top left, real galaxy image from the probes dataset. On top right, an image generated using score-based model. The bottom left plots show the distribution of XPOM for 10 and 100 regions, which as expected follow the correct chi-squared distribution. The botton right plots show the same distributions for the generated images. We see that the chi-squared is close to the expected one, meaning the generated images are nearly indistinguishable from real ones. For these experiments, X XPQM was evaluated by retessellating and resampling the data sets.
between entries and compute distances between the encodings. show the value of X in Fig. 16. We see that X goes down training progresses, until eventually plateauing, as expected.
F.5 ASTROPHYSICS EXPERIMENTS
As an example of PQMass applied to scientific problem, we test the method on problem from astrophysics. We trained a score-based model (SBM) on 256x256 real galaxy images from the Probes dataset, which is compendium of 3163 high-quality local late-type galaxies Stone & Courteau (2019). Since there has been increasing interest in the use of SBMs as bigh-dimensional priors for inference problems, the question of the accuracy (in the statistical sense) of the trained SBM is very important. Using the PQMass metric, we were able to establish the probability that the generated samples come from the same underlying distribution the training samples. On 256x256 images,
21

---

Published as a conference paper at ICLR 2025
0.0125
0.0000
Fleqeeee
Prequency
0.10
0.0075
0.0050
0.05
0.0025
0.0000
5 10 15
20
25
30
1400
Xiga
Figure 18: Example a true dark image (top) and generated dark (middle). The bottom plot shows distribution of XPOM for 10 regions when comparing half of the 6960 ground truth darks with the other half, while the bottom right compares 2000 generated dark samples from the SBM with the full set of the real darks. The two sets elearly out of distribution, the generative model has not learned the true distribution of hot pixels, despite the evident similarity by eye between the ground truth and generated samples. For these experiments, XPOM was evaluated by retessellating and resampling the data sets.
PQMass runs in less than one hour for 1000 evaluations on a single Nvidia A 100 GPU (about 3s for every x2 evaluation). When compare random split of the probes dataset with itself, the XTOM values are distributed according to the expected x² distribution while we can reproduce curve similar to the one presented in Fig. 5 as function of training iteration. The results are shown Fig. 17.
Next, another scientific application of interest is using SBMs to model noise likelihoods as described in Legin al. (2023). In this case, the accuracy of the learned generative model of paramount importance. Here, we used the case of pixel darks (images of pure noise) from the Near Infrared Imager and Slitless Spectrograph (NIRISS) aboard the James Webb Space Telescope (JWST) as the noise to be modeled. This instrument has Gaussian noise well as multiple sources of non-Gaussian additive noise, including dead pixels, faint and saturating cosmic rays, bleeding, a zero bias, blotches, and f/f noise. The lattermost source of noise is especially difficult completely correct for using traditional methods (Albert et al., 2023), effectively reducing the usable information of scientific images, especially for low signal-to-noise ratios. As additional experiment, we have trained SBM on 4000 of the full set of 6960 real 2048x256 pixel darks. We show examples of the ground truth images and generated images Fig. 18 as well as PQMass comparisons, one between two sets of the ground truth and another between the generated images and the ground truth. While the XPoM metric can clearly show that half of the real dark samples are in distribution with the other half (bottom left plot Fig. 18) using 10 regions, can also detect that the generated samples are not in distribution with real dark samples, despite their visually evident resemblance. Upon further investigation, this can be confirmed by the realization that the correct placement of hot pixels has not yet been fully learned by the generative model.
now compare PQMass against other standard metrics in the fields, Feature Likelihood Divergence (FLD, Jiralerspong al. (2023)) and Fréchet Inception Distance (FID, Heusel et al. (2017)). We utilize two astrophysics datasets introduced earlier- Probes and JWST and introduce a third dataset, SKIRT TNG (Botrell et al. (2024)), which consists of simulated galaxy data. We use the SBM trained by Missael Barco et al. (2024). We compare samples generated from the respective SBM against the validation set, the ground truth. Additionally, conduct null test where, for each dataset, split
Proiers
22

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of PQMass, FLD, and FID metrics across various astrophysical datasets. PQMass is configured with 10 regions, yielding an expected score approximately for in-distribution comparisons. For PQMass, scores significantly above 9 indicate outtof--istribution samples. All PQMass experiments were done by retessellating and resampling the sets for evaluating XTOM For FLD and FID, lower scores generally suggest similar distributions. "Samples Ground Truth" compares generated samples to the original dataset, while "Ground Truth vs Ground Truth" compares two subsets of the original data to assess metric consisteney.
Dataset
Dimensionality
Num of Samples
PQMass
FLD
FID
Probes (Samples vs Ground Truth)
3x256x256
1000 vs 2059
10.97
52.13
594.94
Prohes (Ground Truth vs Ground Truth)
3x256x256
1029 vs 1030
9.42
-21.72
84.64
SKIRT TNG (Samples vs Ground Truth)
3x64x64
1500 vs 2549
9.84
24.29
613.74
SKIRT TNG (Ground Truth vs Ground Truth)
3x64x64
1230 vs 1229
9.83
-4.09
12.14
JWST (Samples vs Ground Truth)
Ix256x2048
3480 vs 3480
131.70
11.08
128.59
JWST (Ground Truth vs Ground Truth)
1x256x2048
2000 vs 6969
7.88
-3.06
1.07
the validation set into two subsets to test whether the metrics can detect that the subsets come from the same distribution. We highlight the dimensionality of each dataset, with JWST being the highest dimensional experiment.
We showcase our results in Table 5, where we use 10 regions for all experiments. In the null test, PQMass is consistently able to detect that they come from the same underlying distribution; however, FLD struggles, returning negative values indicating a measure of duplicates in the two datasets, which we know to not be true. FID struggles with Probes and SKIRT TNG but can showcase that for the JWST example, it is in distribution. Furthermore, when comparing samples generated from the SBM to the validation test, PQMass can detect that the samples come from the same distribution for Probes and SKIRT TNG, which we know to be correct. It can also capture the out-of-distribution nature of the JWST samples, which is correct. However, the same cannot be said for FLD FID. For Probes and SKIRT TNG, FLD and FID claim the samples are out of distribution, and for JWST, FLD claims that the samples are in distribution, whereas FID correctly claims that the samples are out of distribution. Here, we showcase that PQMass can detect if the two datasets are in or out of distribution consistently where other standard metrics in the field either fail or are highly inconsistent.
F.6 EXPERIMENT ON PROTEIN SEQUENCES
As another experiment on a real dataset with a different structure, we showcase the ability of PQMass detect differences in datasets of protein sequences. As our example, we use the ESM Metagenomic Atlas dataset (Lin et al., 2023)?. We perform two comparisons: In the first one, we randomly split the dataset into two, and repeat the null test shown in $3.1 and $AA. As our distance metric, we use the Levenshtein, or edit distance (i.e., the number of edits needed to turn one sequence into the other). The result is shown in the blue histogram of Fig. 19. This shows not only that our method can also work in this data format, but also that it can work with different distance metric.
Secondly, we split the dataset into two: One with Protein post-translational modification (PTM) higher than 0.5, and one with PTM lower than 0.5. This is equivalent to splitting the dataset into one of high confidence protein sequences, and one of low confidence. We then repeat the test and show the result in the orange histogram of Fig. 19. We can clearly see that PQMass can detect that these two datasets come from different distributions. It is a non-trivial result that, by comparing edit distances in sequence space, we can detect that the high PTM and low PTM sequences belong to different distributions.
G PQMASS IN DIFFERENT METRICS
The PQMass test is performed using only distances between sample points, thus one may choose any distance metric, or kernel (representation) function, to perform the test. The choice of metric may affect what features PQMass is most sensitive to. In Fig. 20 we show how PQMass performs on toy problem using various distance metrics. A Gaussian mixture model with 20 components is created in 100 dimensions, for each component the means are drawn from uniform distribution with range (-10, 10) in all dimensions, the covariance matrices are constructed with eigenvectors aligned randomly and eigenvalues drawn from a log-uniform distribution with range (-1, 1). The
Phttps://esma las. com
23

---

Published as a conference paper at ICLR 2025
Protein Sequences
Random Split
0.035
PTM Split
Expected
0.030
0.025
Froquency
0.020
0.015
0.010
0.005
0.000
100
200
300
400
500
XPQM
Figure 19: Experiment with protein sequences. In bluc, compare two randomly chosen subsets of the ESM metagenomic Atlas dataset. In orange, we compare high PTM to low PTM sequences. PQMass clearly detect random splits are samples from the same distribution, while high and PQM proteins correspond to two different datasets. This experiment was done by retessellating and resampling the sets for evaluating XPOM
weight for each component is also drawn from log-uniform distribution with range (-1, 1). For each test, we draw two sets of 22 samples from the GMM as input to PQMass. The null test takes these inputs as-is, for the "scale" out-of-distribution test the samples are scaled by 1 1.08, and for "rotated" out-of -itto---iitribttion test the samples are rotated by approximately 0.08 radians on a random axis (scale and rotation chosen arbitrarily for figure quality). To generate the histograms in the figure, the tests are re-run 2 times each.
All metrics Fig. 20 correctly fail to reject the null test (left subplot), as should be expected.
The only "failure mode" for PQMass is to not reject the null when the two distributions are different because the metric is degenerate (i.e., is really a pseudometric). This occurs the centre subplot, where correlation and cosine metrics fail to reject the null because they are not sensitive to rescaling. In the right subplot, where we use a small rotation as the perturbation, the correlation and cosine are, in fact, the most sensitive this change, demonstrating how choice of metric impacts the sensitivity to differences in the two samples.
We also repeat the experiment described in $3.3, where we evaluate the impact of various distance metrics on PQMass as Gaussian noise with increasing variance is added to the ImageNet dataset. In Figure Fig. 21, we show that when no noise is added (null test), all distance metrics will correctly detect that the images are in distribution (left subplot). We also show that as we add noise, different metrics showcase different sensitivity and thus affect XPOM differently. In both tests (and indeed all our tests) the Euclidean distance the one used in the main experiments is among the most discriminating.
H COMPARISON OF PQMASS TO OTHER TWO-SAMPLE TESTS
Here, we compare PQMass to other two-sample tests. One such class of tests is described in Schilling (1986), which proposes a nonparametric approach to determine whether two multivariate samples of points drawn from the same underlying distribution. In the proposed class of tests, the statistic used is calculated as the fraction of nearest-neighbor comparisons where both the point and its nearest neighbor from the same sample. Here, we consider two specific variations: unweighted version,
24

---

Published as a conference paper at ICLR 2025
In distribution
Out of distribution (scaling)
Out of distribution (rotation)
0.030
Cityblock
Euclidean
0.025
Chebyshev
Seuclidean
Canberra
Density
Braycurtis
Comelaaton
0.015
Cosine
x*(99)
0.010
0.005
0.000
7s
100
125
200
400
400
600
x² value
x² value
x² value
Figure 20: Comparison PQMass applied with various metrics. The metrics are described in detail
scipy. . caist documentation; the first three correspond to standard L1, L2, and Linf x² distributions presented in each figure. Left: null test to show any metric will return null result the null case. Middle: out-of-distribution example where samples are scaled by small amount. Right: out-of-distribution example where samples are rotated by small amount.
104
0.035
Braycurtis
Seuclidean
0.030
Euclidean
Cosine
Correlation
Density
0.025
Cityblock
Chebyshev
10
0.015
0.010
0.005
102
0.000
60
80
100
12.
140
160
0.0
0.2
0.4
0.6
D.8
1.0
Noise level
Figure 21: Comparison of the performance of various distance metrics to detect the addition of Gaussian noise with increasing variance noise ImageNet data. Left: Null test with noise added, where all metrics correctly identify the data as in-distribution. Right: Gaussian noise with progressively larger variance added to one of the sets affects XTOM differently depending on the metric used. For this experiment, XPOM retessellating and resampling the datasets.
evaluated by
25

---

Published as a conference paper at ICLR 2025
Table 6: Two multivariate two-sample tests from Schilling (1986) (unweighted and weighted) applied to two sets of samples from N 1) and ( 1). We wish to assess whether these tests can detect that two samples are out of distribution. For each test, we display the expected in in-distribution value and the obtained result, as well as the corresponding p-value. In both cases, the results are statistically very close to the expected values, indicating no detection that the two distributions out of distribution.
Metric
Expected Result
Actual Result
p-value
TE,
49,4900
52,4000 1.1245
0.5596
Uk,n,w
1.1300
0.7055
0.30-
5-
0.08
Proqaccey
0.06
3-
0.00
0.02
0.00
20
0.5
0.6
0.8
X
p-tulae
Figure 22: Comparison of samples from N(O, 1) to 50 samples from NO0.5, 1), with PQMass. Unlike the multivariate two-sample test shown Table 6, PQMass can confidently detect two sets of samples are indeed not from the same distribution, showcasing that PQMass is more sensitive than the multivariate two-samplc tests presented in the main text.
denoted Tk.n: which evaluates the proportion of nearest neighbors that are similar, and a weighted version, denoted Uk.n.w, where distances are assigned decreasing weights with increasing distance.
We show that PQMass can outperform this methodology by comparing the performance of the unweighted and weighted nearest-neighhor two-sample tests against PQMass. Specifically, we consider two distributions, I), and N00.5, I), where dimensional mean vectors and denotes the 2x2 identity matrix. Samples from these distributions should be identified as -utoofddistributinn relative to each other. Using 50 samples from each distribution, we evaluate Tr., and Un.n. in Table 6. For the unweighted test statistics, the expected value is Tk., 49.4900, and the observed value is Tk., 52.4000, with p-value of p value,,,, 0.5596. These results indicate that the unweighted wamslm test fails to reject the null hypothesis and incorrectly suggests the distributions are in-distribution. For the weighted two-sample test, the expected Uk..,w 1.1300 1300, the observed value is Uk,,,w 1.1245, and the p-value is p-value 0.7055. Again, the results do not indicate a significant difference between the distributions. In contrast, as shown in Fig. 22, when running with PQMass using 10 regions (due to limited samples), the resulting XFQM as well the p-valuepoM- confidently indicate the data does not come from the same underlying distribution.
(0, 0) and 0.5 = (0.5, 0.5) are 2-
I PERMUTATION TESTS
When PQMass run for a single iteration (single tessellation) is an exact statistical test, however the power of this test is considerably dependent on the particular arrangement of the Voronoi bins. To increase the power of the test, we repeat the test multiple times and verify that XłoM is x² distributed. In cases where data samples are limited, however, running multiple tests by ressampling and retessellating can be difficult. In those cases, in particular in some of our experiments, we can run many retessellations on the same sets of samples to maximize the use of existing samples and ensure encounter especially discriminating configurations.
In this case, however, since the samples in the retessellations are the same, there is some shared information between tests and the XPOM values from these retessellations must no longer follow x distribution under the null hypothesis. This presents challenge for understanding the meaning of
26

---

Published as a conference paper at ICLR 2025
Nuil
dictribution (pvolue
perratettas 21191
0.000
1ert str
6,775
6,125
6,77
6.033
50
340
306
320
142
rensopllatioed
Figure 23: Permutation test for validity of taking the mean of many PQMass retessellations. Samples are drawn from 100 dimension standard Gaussian, we use 1000 samples for each input. Shown are the x x²(9 distribution (black dashed), mean of many retessellations (red vertical line), and the permutation test (purple histogram). Left: A null test which, as expected, fails to reject the null hypothesis, Right: An out of distribution test where one sample has been scaled by 1.1. The null hypothesis is indeed rejected.
PQMass x² retessellation histogram. This issue can be easily eliminated via permutation test (Good, 2013).
Any measurement on some dataset can be turned into an exact test on the null hypothesis that two samples are drawn from the same distribution using a permutation test. In brief, permutation test involves taking some scalar measurement on two datasets, lets say PQM(x.)) y) where x and are the two samples, and we re-run the measurement on permutation of the datasets PQM(x', y') where x' and y' are random permutation of the combined dataset (x, y). The permuted samples are thus drawn (without replacement) from a mixture distribution of the two samples. Running many permutations one builds up a distribution of values for PQM(x', y') against which we may compare the original value PQM(x. y) to find p-value. At an intuitive level, this test determines how unique the original arrangement of y samples are under the measurement PQM.
The PQMass test is especially sensitive to the differences in two samples, making an ideal choice for such test. Specifically, we may now take PQMass run with many retessellations, take the mean of the reported x² values, and use this mean as our measurement on the two samples; now fully utilizing the discriminating power of the retessellations in an exact test. We perform a permutation test using PQMass in Fig. 23. The histogram shows PQMass run on random permutations of the input samples ("permute stats" in the figure) while the vertical line shows the result for the input samples (" ("test stat" in the figure). In the null case, we achieve a p-value consistent with (O, 1) while in the -iitr---ditriuutionn case we reject the null with low p-value.
In both cases, the mean of many re- re-retessellations on the original sample would have placed us well within the x²(99) distribution, showing how the permutation test is more powerful at rejecting the null, if such extra power is needed.
In practice, find that, for high-dimensional problems, the retessellations without resampling very closely approximate x² distribution (in the null case), though this is not the case in low dimensions. Since is common in high-dimensional spaces for PQMass be highly discriminating. it is often not necessary to perform the permutation test to reject the null; one can conservatively do so using the full x2 distribution.
If, after a few retessellations, one finds that they cannot easily reject the null, then performing a permutation test can add more discrimination power at the cost of extra compute. Even with this extra computational cost, PQMass remains computationally feasible on large and high-dimensional datasets. We note again that with permutation test it is possible to turn any metric on two samples into an exact two-sample test; for example, the energy distance may be used (Székely et al., 2004), but the computational complexity of that test is higher than PQMass.
27


Reviews:
[{"review_id": "19KAQsE4Ct", "reviewer": "Reviewer_YLXe", "summary": "The paper introduces PQMass, a likelihood-free approach for assessing the quality of generative models by comparing sample distributions without estimating probability densities. PQMass operates by dividing the sample space into non-overlapping regions, using chi-squared tests to measure the similarity between the distributions of real and generated samples in these regions. The method yields a p-value indicating whether samples from two sets arise from the same distribution. Authors argue that PQMass offers several advantages: it does not require auxiliary models or feature extraction, works across various data types, and scales to moderately high-dimensional data. Through experiments, authors aim to show that PQMass is effective in evaluating fidelity, diversity, and novelty in generated samples, providing a robust alternative for assessing generative models in machine learning.", "questions": "I would like to hear authors' opinions about the detailed weaknesses of their proposal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- PQMass introduces a new, likelihood-free approach for evaluating generative models by comparing sample distributions without density estimation or feature extraction, which is a fresh alternative to traditional metrics like FID and MMD.\n\n- The experiments are comprehensive, testing PQMass across various generative models, data types, and dimensions. The paper includes comparisons with established metrics, scalability tests, and ablation studies, which aims to show PQMass’s robustness, scalability, and ability to assess fidelity, diversity, and novelty effectively.\n\n- PQMass offers a potentially practical, scalable solution for evaluating fidelity, diversity, and novelty in generative models across diverse, high-dimensional data domains, filling relevant gaps in current model evaluation techniques.", "weaknesses": "- The use of L2 or L1 distance metrics may limit PQMass's effectiveness, especially when the data resides on a complex manifold. These metrics may not capture meaningful differences in such cases, potentially leading to inaccurate assessments. \n\n- Experimental evaluation mostly focus on synthetic data or standard datasets like MNIST and CIFAR-10. Testing on more complex, real-world datasets would strengthen PQMass’s claims about the performance of the proposal.\n\n- Although the paper briefly addresses other data types, it primarily evaluates PQMass on image data. Expanding experiments to more complex, structured data (e.g., text or biological data) and analyzing the impact of different distance metrics in these contexts would help to better validate PQMass performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hFmhZRTEUe", "reviewer": "Reviewer_6iXh", "summary": "The paper proposes a method for testing for the equality of two distributions from which two samples are obtained, by partitioning the sample space and comparing the multinomial vectors of counts of samples in each of the partitions via a chi-squared-distributed test statistic. The approach works for broad classes of data.\n\nEDIT: Following the review/discussion period and (prospective) changes to the manuscript by the authors, I am increasing my score by 1.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well presented and understandable. A broad range of examples and simulations were presented demonstrating fairly broad applicability. The test statistic is a basic and well known statistical quantity, so the method is fast to implement.", "weaknesses": "Not clear if this is a weakness or a strength, but the proposed test is a well known, basic statistical test. The contribution of this paper, therefore, is to note that its easy to apply this test to a broad range of data types in order to detect a difference between the generating distribution of two samples. It's not clear if this is substantial enough a contribution for ICLR, though I do not wish to imply that substantial contributions need to involve complex methods.\n\nConsistency Guarantees:\nI'm not quite clear on or convinced about some of the claims being made.\n* a) Proposition 1: Does the consistency here require any conditions on m and n as they approach -> infinity? E.g., what about their relative convergence rates (does m->infinity faster than n, for example). E.g. what if m/n -> K, where K=infinity, a finite constant, or zero. Presumably K=infinity or K=0 would break things and the test would no longer work (especially as you're sampling finite n_R samples).\n* b) Proposition 1: Some clarification on the claim also needed here, as it's conditional on the n_R samples. E.g. if p = t_{\\nu_p} and q = t_{\\nu_q} (univariate t distributions with different degrees of freedom), and z_1 = -z_2, then you will never distinguish p and q under any rates of m, n -> infinity.\n* c) Proposition 2: This proposition is ok, but really this situation won't happen, as the method explicitly requires that the z_i are taken from the set of (x_i, y_j), and so n_R is bound by m+n which are fixed and finite in this proposition (though the method will fail far before this as there will be no samples left to calculate the test statistic), as noted in the discussion. So how useful is Prop 2 in practice, particularly for small m+n?\n* d) Proposition 2: \"no information is lost by the PQMass statistic.\" would seem to be a bit of an overextension in light of the above point. But also because for n_R->\\infty, the distribution of the statistic under the null would be a chi-squared distribution with infinite degrees of freedom. \"No information is lost\" only happens in this limit (as otherwise with a finite n_R, information can clearly always be lost), but how does your test statistic compare to its null distribution in this limit?\n\nPractical implementation comments:\n* a) As far as I can tell, you've only considered exactly balanced samples each time: m=n. While in some applications the user has control over both m and n, this will not be the case in many scenarios. So how well does the test perform when the two samples are imbalanced, including extreme cases?\n* b) Using reference samples z_i randomly taken from the two distributions (x_i) and (y_i) has a long history, even when comparing samples from two MCMC samplers (or the same MCMC sampler twice, when considering convergence diagnostics; e.g. Sisson & Fan (2007). Statistics and Computing, 17, 357-367). So this approach is fine. However, a disadvantage of this is that when p and q differ only, say, by a small mode with low weight in the tail, then by sampling uniformly over the x_i and y_j, one is reasonably likely to miss samples in this mode, as the sampling will naturally favour repeated samples in areas of higher density, and hence the test will miss the obvious difference between p and q. (The examples in Section 3 seemed to consider equally weighted mixtures only ...). Hence the need to repeat the test multiple times with different z_i. Is there any argument to be made to reconsider this equal sampling of the z_i from (x_i) and (y_j), to ensure there is a greater spread over the space, rather than doubling (and tripling, and ...) down on high density regions?\n* c) Figure 2: presumably when \"this process\" is repeated 500 (or 200?) times, this also means resampling the z_i? (Could be clearer in the caption.) But really, this histogram doesn't really seem to follow the red density. Suggest that the number of reps is increased here considerably if you're wanting to claim these distributions are the same. Also the y-axis label of \"Frequency\" seems to be incorrect, as the axis ticks aren't measured in counts.\n* d) In the results in general, the empirical mean of the PQMass statistic (4) seems to be taken as \"the statistic\" for determining whether p=q (or p!=q). Is it obvious that this mean statistic has a chi-squared(n_R) distribution under the null? Each of the components of this statistic are chi-squared(n_R) (conditional on z_i) under the null, but is it obvious that they are independent from eachother? (chi-squared(m) + chi-squared(n) = chi-squared(m+n) if the two LHS distributions are independent). Certainly the alternative distributions are not the same for different z_i. So can this \"mean PQMass statistic\" actually be compared to the chi-squared null distribution in any meaningful way? Or is the use of the mean PQMass statistic merely limited to relative value comparisons? In which case, the paper should be clearer about all of this and tone down the chi-squared relevance.\n* e) Section 4: Limitations. The method also appears to require independent samples (x_i), (y_i), otherwise the multinomial model does not hold. How likely is this in practice? In the MCMC comparisons in Table 1? Etc.\n\n\nTypos:\n* p.2 para 2, wrong parentheses surrounding reference list starting with Theis et al.\np.5. Algorithm step 1. \"choice\" -> \"chosen\"\np.6. Section 3.1. The number of test replicates here is 200, but in Figure 2 it's listed as 500. One of these must be wrong.\np.6. Table 1. D was previously the distance metric between two sample points (c.f. Section 2.2, Algorithm). Now D has switched to number of dimensions. Maybe separate the notation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTjtv8jOW9", "reviewer": "Reviewer_mbPi", "summary": "The authors proposed a simple method for comparing the distribution $q$ of a generative model with the true data distribution $p$. The proposed method first discretizes the data space and then compare the proportion of data points in each region. Since this becomes a distribution comparison for a multinomial distribution, it can be tested using Pearson’s $\\chi^2$ test. If the test indicates a difference between the two distributions, one can conclude that $p \\neq q$, making this method a sufficient condition for identifying distribution differences. The authors demonstrated the effectiveness of the proposed method on synthetic data as well as datasets such as MNIST and CIFAR-10.", "questions": "What are the strengths or advantages of the proposed method compared to other distribution comparison techniques beyond MMD and W2?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The strength of the proposed method is in its simplicity. After discretizing the data space, one only needs to count the data points in each region and calculate the test statistic. This makes the method computationally much more efficient compared to existing distribution comparison methods, such as MMD.", "weaknesses": "A weakness of this study is the limited comparison with other two-sample test methods. The authors simplified the problem by dividing the data space and reducing it to a multinomial distribution comparison. However, various other methods have been proposed for distribution comparison not only MMD and W2. For example, [Ref1] explores distribution comparison using classifiers and its application to GAN evaluation, while simpler approaches, like those using nearest neighbors, are also available [Ref2]. Although the simplicity of the proposed method is an advantage, comparing its performance with these existing distribution comparison methods would be essential for evaluating its practical utility.\n\n* [Ref1] Revisiting Classifier Two-Sample Tests, ICLR 2017.\n* [Ref2] Multivariate Two-Sample Tests Based on Nearest Neighbors, Journal of the American Statistical Association, 1986.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vk48Vih88C", "reviewer": "Reviewer_b1WN", "summary": "The article proposes a new method for comparing samples from two distributions, providing a hypothesis test for whether they are identically distributed. The method introduces a Voronoi tessellation, dividing the data space into non-overlapping regions, and then assumes a multinomial likelihood describing the prevalence of observed data in each of the regions: this likelihood is then used to develop a hypothesis test as to whether each distribution has the same underlying probabilities of generating data in each region, which then implies equivalence of the distributions with some assumptions. Some theoretical results are derived to strengthen this argument. The method is then tested on several empirical problems.", "questions": "Methodological questions:\n\nIs there any particular reason that you pursued Pearson’s chi squared test ahead of Fisher’s exact test, which should be less vulnerable to small contingency cell counts, which could easily happen here with the randomised voronoi tessellation?\n\nChoice of centres:\n\nIs it really hard to do better than random? Some sort of relatively simple diversity measure would potentially get you quite far here.\n“Sample the reference points from a uniform mixture of the empirical distributions of samples from p and q”\n\nWhat if there are more samples available from one distribution?\n\nWhat if one of the distributions is known to be more accurate representation of reality and therefore give a better coverage of realistic regions of probability space?\n\n“Perform the test multiple times”? How many is necessary? What are the computation implications?\n\n“This effectively marginalizes the choice of tessellation”: this comment makes me wonder whether a Bayesian formulation of the problem would work well, since you are effectively having sample the latent allocation of the points to reference centres vs cell populations. There will be analytical conjugate Bayes solutions to the multinomial distribution comparison too, so there should be no more pesky posterior sampling in addition to what is being done already.\n\nData balance between distributions:\n\nWhat if there are unbalanced numbers of samples available from each distribution? This will be quite common, since in general distributions are not equally easy to sample from. This is especially true if comparing simulated data and real data, or complex vs simpler models. How would this influence the sampling of reference points?\n\nLimitations: good for you for being open “if the number of samples is too small”. Do you have any idea how small? Does it scale with p in a demonstrable way?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is pleasingly simple for a relevant contemporary challenge in stats/ML, and explained pretty clearly. The experimental results are strong and quite convincing, especially in higher-dimensional problems without an obvious feature representation.\nThe figure quality is high throughout, which is a good thing in general.\nThe experiments themselves are pretty thorough, as are the descriptions of the experiments, especially the astrophysics experiments that might be hard for a non-expert audience. The example detecting the small additive noise is nice in Appendix F.2. Demonstrating the method can find an effect invisible to the naked eye is good.", "weaknesses": "The very high figure quality has the probably unintended consequence that the resulting pdf file size is very large (>17MB), which makes it kind of a pain to handle by conventional means. If there were some way to reduce this without overly compromising readability, then that would be good. I suspect that you have one or two figures in the appendix that are unnecessarily high resolution when printed out, but I haven’t figured out which ones (possibly figures 17/18).\n\nA few small language issues:\n\n“Miscrepancy” line 356. I assume you meant “discrepancy”? English in general pretty good.\n“The instrument has multiple sources of non-Gaussian additive noise, including… Gaussian noise” is kind of odd.\n\nFigure design:\n\nFigure 6 is hard to understand immediately. The claim about “strongly correlate” (line 446) does not jump out intuitively from the data plotted.\n\nSome of the descriptions of the method’s properties are a little overly flattering:\n\nYou call this a “Likelihood-free method”... but, you do use a multinomial distribution.\nDesiderata 1: “PQMass works with the intervals of pdfs, it does not make assumptions about the true underlying density field”. The cdf is defined by the pdf, and assumptions about one will imply assumptions about the other, so I don’t think this argument is particularly strong.\n\n“These results show that, at least for “well-behaved” densities, no information is lost by the PQMass statistics”\nDid you not say at the start that you didn’t make assumptions about the underlying density?\n“No information is lost” in what sense? Clearly information is lost when you quantise. Is this claim meant in some asymptotic limit?\n\nMore explanation of the 2-d Gaussian mixture model and the 10-d funnel distribution would be good rather than just a reference, if you can find the space. Appendix is always an option.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)