PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation

## Abstract

Abstract We propose a comprehensive sample-based method for assessing the quality of generative models. The proposed approach enables the
estimation of the probability that two sets of samples are drawn from the same distribution,
providing a statistically rigorous method for assessing the performance of a single generative model or the comparison of multiple competing models trained on the same dataset. This comparison can be conducted by
dividing the space into non-overlapping regions and comparing the number of data samples in each region.
The method only requires samples from the generative model and the test data. It is capable of functioning directly on high-dimensional data, obviating the need for dimensionality reduction. Significantly, the proposed method does not depend on assumptions regarding the density of the true distribution, and it does not rely on training or fitting any auxiliary models. Instead, it focuses on approximating the integral of the density (probability mass) across various sub-regions within the data space.

## 1 Introduction

Generative modeling has become an important and ubiquitous topic in machine learning. The goal of generative modeling is to approximate a distribution given a set of samples from the distribution.
Generative machine learning has witnessed the development of a succession of methods for distribution approximation, including variational autoencoders  (VAEs, Kingma & Welling, ), generative adversarial networks  (GANs, Goodfellow et al., ), normalizing flows (Rezende & Mohamed, ), and score-based (diffusion) generative models (Ho et al., ).

With advancements in generative models, evaluating their performance using rigorous, clearly defined metrics and criteria has become increasingly essential. Disambiguating true from modeled distributions is especially pertinent in light of the growing emphasis on AI safety within the community,
as well as in scientific domains where stringent standards of rigor and uncertainty quantification are needed for the adoption of machine learning methods.
When evaluating generative models, we are interested in three qualitative properties (Stein et al., ; Jiralerspong et al., ):
Fidelity refers to the quality and realism of individual outputs generated by a model. It assesses how indistinguishable each generated sample is from real data. Diversity pertains to the range and variety of outputs a model can produce. For example, a model that misses a mode in the training data, producing no samples of the given mode, has lower diversity. Novelty refers to the ability of a model to generate new, previously unseen samples that are not replicas of the training data yet are coherent and meaningful within the context of the task. Essentially, it means appropriately interpolating the regions in between the training data in higher-density regions.

There have been two classes of methods for evaluating generative models: sample-based methods, which compare generated samples to true samples directly, and likelihood-based methods, which make use of the likelihood of the data under the model. It has been observed that likelihood-based methods, such as data negative log-likelihood (NLL), show too much variance to be useful in practice and are ‘saturated’ on standard benchmarks, i.e., they do not correlate well with sample fidelity, particularly in high-dimensional settings (Theis et al., ; Nalisnick et al., ; ; Yazici et al., ; Le Lan & Dinh, ). On the other hand, most sample-based methods cannot measure the combination of fidelity, diversity, and novelty. For example, the Fréchet Inception Distance (FID; Heusel et al., ) and the Inception Score (IS; Salimans et al., ; Sajjadi et al., ) measure fidelity and diversity, but not novelty, while precision and recall (Salimans et al., ) measure only fidelity and diversity, respectively; and authenticity (Alaa et al., ) measures only novelty. Recently, the Feature Likelihood Score (FLS; Jiralerspong et al., ) was proposed as a measure that captures fidelity, novelty, and diversity. However, FLS essentially relies on the approximation of the underlying distribution by a Gaussian mixture model and consequently requires feature extraction and compression for high-dimensional models using an auxiliary model.

In this work, we propose PQMass, a general statistical framework for evaluating the quality of generative models that measures the combination of fidelity, diversity, and novelty and scales well to high dimensions without the need for dimensionality reduction. PQMass allows us to accurately estimate the relative probability of a generative model given a test dataset by examining the *probability mass* of the data over appropriately chosen regions.

The main idea motivating PQMass is that given a sub-region of the entire space, the number of data points lying in the region follows a binomial distribution with a parameter equal to the region’s true probability mass (which is unknown and should be marginalized). By extension, given a partition of the space into regions, the counts of points falling into the regions follow a multinomial distribution. PQMass uses two-sample tests on these multinomial distributions’ parameters for prudently selected regions defined by Voronoi cells. This approach gives us a p-value, measuring the quality of a generative model (the probability of a model given the training data). It also allows the comparison of various competing models by analyzing the ratio of their probabilities.

Since PQMass works with the *integrals* of the true and estimated probability density functions over regions, it does not make any assumptions about the true underlying density field. PQMass is also computationally efficient and scales well with the dimensionality of the data, allowing us to directly evaluate generative models of high-dimensional data, such as images, without the need for feature extraction or compression. Finally, PQMass is general and flexible, allowing the evaluation of generative models on any type of data, including images and time series.

Our contributions are the following: In § 2, we introduce the theoretical framework for PQMass. In § 3, we introduce an algorithmic instantiation of the framework. In § 4, we present experiments on synthetic and real-world datasets and compare PQMass with prior methods.

## 2 Theoretical Framework

Figure: Figure 1: An illustration of the statistic computed by PQMass. Left: We start with two sets of samples $\left\{x_{i}\right\}_{i=1...n}\sim p(x)$ and $\left\{y_{i}\right\}_{i=1...m}\sim q(y)$, shown as red and blue points, respectively. For a given region $R\subseteq X$, the fraction of samples in $R$ for each set of samples follows a binomial distribution, with parameters $n$ and $\mathbb{P}_{p}(R)$ for the red samples, and $m$ and $\mathbb{P}_{q}(R)$ for the blue samples. We can use this binomial distribution to estimate the probability that both binomial distributions are the same. Right: If we can build non-overlapping regions $\left\{R_{i}\right\}_{i=1...n_{R}}$, for example, using a Voronoi tesselation, the distribution of samples in the regions follows a multinomial distribution, for each set of samples. PQMass performs hypothesis tests on equality of the two multinomial distributions to obtain a p-value.
Refer to caption: x1.png

Our problem statement is the following. We have two sets of samples $\left\{x_{i}\right\}_{i=1...n}\sim p(x)$ and $\left\{y_{i}\right\}_{i=1...m}\sim q(y)$, where both $p$ and $q$ have support in some region $X\subseteq\mathbb{R}^{d}$. We are interested in testing the statistical hypothesis that $p=q$, that is, the two samples came from the same distribution.

### 2.1 Measuring Equivalence

We recall an elementary fact from the foundations of probability: two probability distributions $p(x)$ and $q(x)$ over a measurable region $X\subseteq\mathbb{R}^{d}$ are equivalent if they are equal as measures, i.e.,

$$ $\mathbb{P}_{p}(R)=\mathbb{P}_{q}(R)\qquad\text{$\forall R\subseteq X$, $R$ measurable}.$ (1) $$

Following this fact, we can measure whether two distributions are equivalent by estimating, using samples drawn from these distributions, whether the mass they assign to a large number of finite, arbitrary regions is equal.

We will now introduce a method for estimating this probability. We start with the following simple observation:

###### Lemma 2.1 .

Given a set of samples $\left\{x_{i}\right\}_{i=1...n}\sim p(x)$, the probability in a region $R\subseteq X$ can be approximated by the proportion of samples in $R$:

$$ $\mathbb{P}_{p}(R)=\mathbb{E}_{x\sim p(x)}\left[\mathds{1}\left(x\in R\right) \right]\approx\frac{1}{n}|\{i:x_{i}\in R\}|.$ (2) $$

This estimate is unbiased, and in the limit of infinite samples, this approximation becomes exact (i.e., the estimator is consistent).

###### Proof.

We start by defining the random variable $X$ as

$$ $X=\begin{cases}1&\text{if }x\in R\\ 0&\text{otherwise}\end{cases}$ (3) $$

We have that

$$ $\mathbb{E}_{x\sim p(x)}\left[X\right]=\mathbb{P}_{p}(R)$ (4) $$

We can approximate this expectation by the empirical mean of the samples:

$$ $\mathbb{E}_{x\sim p(x)}\left[X\right]\approx\frac{1}{n}\sum_{i=1}^{n}X_{i}= \frac{1}{n}\sum_{i=1}^{n}\mathds{1}\left(x_{i}\in R\right)$ (5) $$

which is the proportion of samples in $R$.
∎

In the following, we will use the notation

$$ $k({\bf x},R)\equiv\sum_{i=1}^{n}\mathds{1}\left(x_{i}\in R\right).$ (6) $$

Here, $k({\bf x},R)$ is the number of samples of the entire dataset, ${\bf x}$, in a region $R$. We can approximate the probability mass in a region $R$ by the proportion of samples in ${\bf x}$ that lie in $R$. We can now use this approximation to measure whether two sets of samples are equivalent, by measuring whether the proportion of samples in a large number of finite regions is the same for both sets of samples.

A simple, non-probabilistic method for assessing the equivalence of two datasets might involve directly comparing their respective proportions. Nonetheless, this paper will demonstrate a framework for conducting a more comprehensive analysis. This approach aims to quantify the probability that the two datasets are equivalent, thereby providing a rigorous statistical measure of their similarity.

###### Lemma 2.1 .

###### Proof.

### 2.2 Quantifying the Probability in a Region: A Frequentist Approach

We will now introduce a method for measuring this proportion. First, we notice the following important property:

###### Lemma 2.2 .

Given a set of samples $\left\{x_{i}\right\}_{i=1...n}\sim p(x)$, the number of samples in a region $R\subseteq X$ follows a binomial distribution:

$$ $k({\bf x},R)\sim\mathcal{B}\left(n,\mathbb{P}_{p}(R)\right)$ (7) $$

This means that, given two sets of samples $\left\{x_{i}\right\}_{i=1...n}\sim p(x)$ and $\left\{y_{i}\right\}_{i=1...m}\sim q(y)$, we can estimate the value of $\mathbb{P}_{p}(R)$ and $\mathbb{P}_{q}(R)$, using $\hat{p}_{R}\equiv k({\bf x},R)/n$ and $\hat{q}_{R}\equiv k({\bf y},R)/n$ as estimators respectively.

We can easily extend this result to any number of non-overlapping regions $\left\{R_{i}\right\}_{i=1...n_{R}}$, by noticing that the number of samples in each region follows a multinomial distribution:

$$ $\left\{k({\bf x},R_{i})\right\}_{i=1...n_{R}}\sim\mathcal{M}\left(n,\left\{ \mathbb{P}_{p}(R_{i})\right\}_{i=1...n_{R}}\right).$ (8) $$

Therefore, given a set of regions $\left\{R_{i}\right\}_{i=1...n_{R}}$, we can estimate the probability that both sets of samples are equivalent by measuring whether the multinomial distributions are the same. We first discuss how to approach this problem from a frequentist perspective. We will discuss the Bayesian approach in the next subsection.

In the frequentist approach, there are multiple ways to measure whether two multinomial distributions are the same (see, e.g., Anderson et al., ; Zelterman, ; Plunkett & Park, ; Bastian et al., ). In this paper, we use one of the simplest methods, the Pearson $\chi^{2}$ test (Rao, ). We will explore alternative methods in future work. For reviews on this test, see (Rao, ). We start by defining the expected counts for each region:

$$ $\hat{N}_{x,i}\equiv n\hat{p}_{R_{i}},\qquad\hat{N}_{y,i}\equiv m\hat{p}_{R_{i}},$ (9) $$

where:

$$ $\hat{p}_{R_{i}}\equiv\frac{k({\bf x},R_{i})+k({\bf y},R_{i})}{n+m}.$ (10) $$

We can then define the $\chi^{2}_{\rm{PQM}}$ statistic using the definition of the Pearson chi-squared:

$$ $\chi^{2}_{\rm{PQM}}\equiv\sum_{i=1}^{n_{R}}\left[\frac{(k({\bf x},R_{i})-\hat{ N}_{x,i})^{2}}{\hat{N}_{x,i}}+\frac{(k({\bf y},R_{i})-\hat{N}_{y,i})^{2}}{\hat {N}_{y,i}}\right].$ (11) $$

This statistic follows a $\chi^{2}$ distribution with $n_{R}-1$ degrees of freedom. Therefore, we can calculate the p-value of the test as

$$ $\text{p-value}(\chi^{2}_{\rm{PQM}})\equiv\int_{-\infty}^{\chi^{2}_{\rm{PQM}}} \chi^{2}_{n_{R}-1}(z)dz.$ (12) $$

This p-value provides an estimate of the probability that the two sets of samples are equivalent.

With that, we can summarize this section into the main theoretical result of this paper:

Given any sampling distribution, or generative model, if two sets of samples are generated from the same distribution, then the statistic $\chi^{2}_{\rm PQM}$ follows a chi-square distribution with $n_{R}-1$ degrees of freedom.

We can then use this result to quantify how different the two sets of samples are.

###### Lemma 2.2 .

### 2.3 Quantifying the Probability in a Region: A Bayesian Approach

We can also use a Bayesian approach to quantify the probability in a set of regions. Once again, we know that the proportion of samples in a set of regions $\left\{R_{i}\right\}_{i=1...n_{R}}$ follows a multinomial distribution. Therefore, we can use the closed-form likelihood of the multinomial distribution to calculate the probability of the proportion of samples in a region $R$ given a set of samples $\left\{x_{i}\right\}_{i=1...n}\sim p(x)$:

$$ $\mathcal{L}_{x}\equiv\mathbb{P}(\left\{x_{i}\right\}_{i=1...n}|\mathbb{P}_{p}, \left\{R_{i}\right\}_{i=1...n_{R}})=\\ {n\choose k({\bf x},R_{i}),...,k({\bf x},R_{n_{R}})}\prod_{i}^{n_{R}}\mathbb{P }_{p}(R_{i})^{k({\bf x},R_{i})},$ (13) $$

and similarly for $\mathcal{L}_{y}$. In a Bayesian approach, we can use the posterior predictive distribution (PPD) to calculate the probability of finding the number of samples, $k({\bf y},R)$, in a region $R$ given the number of samples from the other set, $k({\bf x},R)$, in the same region:

$$ $\mathbb{P}(\left\{y_{i}\right\}_{i=1...m}|\left\{x_{i}\right\}_{i=1...n},\left \{R_{i}\right\}_{i=1...n_{R}})=\\ \int{\mathrm{d}}\lambda_{1}{\mathrm{d}}\lambda_{2}...{\mathrm{d}}\lambda_{n_{R }}\mathbb{P}(\left\{y_{i}\right\}_{i=1...m}|\left\{\lambda_{i}\right\}_{i=1... n_{R}},\left\{R_{i}\right\}_{i=1...n_{R}})\\ \cdot\mathbb{P}(\left\{\lambda_{i}\right\}_{i=1...n_{R}}|\left\{x_{i}\right\}_ {i=1...n},\left\{R_{i}\right\}_{i=1...n_{R}})$ (14) $$

Assuming a uniform prior $\lambda_{i}\sim\mathcal{U}(0,1)$, we can rewrite the integrand as a product of two likelihoods:

$$ $\mathbb{P}(\left\{y_{i}\right\}_{i=1...m}|\left\{\lambda_{i}\right\}_{i=1...n_ {R}},\left\{R_{i}\right\}_{i=1...n_{R}})=\\ \mathbb{P}(\left\{x_{i}\right\}_{i=1...n}|\left\{R_{i}\right\}_{i=1...n_{R}})^ {-1}\\ \cdot\int{\mathrm{d}}\lambda_{1}{\mathrm{d}}\lambda_{2}...{\mathrm{d}}\lambda_ {n_{R}}\mathbb{P}(\left\{y_{i}\right\}_{i=1...m}|\left\{\lambda_{i}\right\}_{i =1...n_{R}},\left\{R_{i}\right\}_{i=1...n_{R}})\\ \cdot\mathbb{P}(\left\{\left\{x_{i}\right\}_{i=1...n}|\lambda_{i}\right\}_{i=1 ...n_{R}}\left\{R_{i}\right\}_{i=1...n_{R}})$ (15) $$

Given that we have a closed-form likelihood, it is possible to calculate the integrand numerically. This is still, however, an integral over variable $\lambda$ with a dimensionality equal to the number of regions. One can numerically approximate this integral using Monte Carlo integration or methods such as nested sampling  (Skilling, ; Buchner, ; Lemos et al., ).
We show an experiment using this Bayesian approach with a low number of regions in § B, but otherwise, given the numerical cost of this method, we will focus on the frequentist approach in this paper, and leave the Bayesian approach for future work.

## 3 Algorithmic Instantiation

Figure: Algorithm 1 The frequentist version of our algorithm

We now introduce an algorithmic instantiation of our framework. The key element in the algorithm is the choice of the regions $\left\{R_{i}\right\}_{i=1...n_{R}}$. We follow  Lemos et al. () in using the Tests of Accuracy with Random Points (TARP) framework. We combine definition 7 and remark 2 of (Lemos et al., ) in the following way:

###### Definition 3.1 .

Given a distance metric $d:U\times U\rightarrow R$, every pair of points $\left(x_{c},x_{r}\right)\sim U$ defines a TARP region as:

$$ $\left\{x\in U\ |\ d(\theta,x_{c})\leq d(x_{r},x_{c})\right\}$ (16) $$

In this case, however, we cannot randomly select regions from $U$. The reason for that is that the derivation of § 2.2 assumes that the different regions do not overlap, which will not be the case if we randomly select regions from $U$. Therefore, we need to generate regions that do not overlap. We do this by noticing the following:

###### Lemma 3.2 .

A set of distinct points $\left\{x^{\rm ref}_{i}\right\}_{i=1...n}$ defines a set of non-overlapping TARP regions, through the Voronoi tessellation of the space $U$.

$$ $R_{i}\equiv\left\{x\in U\ |\ d(x,x^{\rm ref}_{i})<d(x,x^{\rm ref}_{j})\forall i \neq j\right\}$ (17) $$

i.e., the region $R_{i}$ is defined by the points that are closest to the reference point that defines that region, $x^{\rm ref}_{i}$, than to any other reference point.

The $R_{i}$ partition the entire space into non-overlapping regions.(^1^11To be precise, there exists a set of points that are equidistant to two or more reference points. In this case, we could assign the point to the region of the reference point with the lowest index. However, this is an insignificant point, as the boundaries of the regions have measure zero if the points are drawn from a continuous distribution.) We show an illustration of the Voronoi tessellation in the right panel of Fig. 1, and the frequentist version of our algorithm in Algorithm 1.

###### Definition 3.1 .

###### Lemma 3.2 .

## 4 Experiments

### 4.1 Null Test

Figure: Figure 2: Null test. We generate two sets of samples from the same Gaussian mixture model in $100$ dimensions. We then measure the $\chi^{2}_{\rm PQM}$ value of our test ($n_{R}=100$) and repeat this process $500$ times, resampling every time. We can see that the $\chi^{2}_{\rm PQM}$ value is distributed as a chi-squared distribution with $n_{R}-1$ degrees of freedom, as expected.
Refer to caption: x2.png

We start by testing PQMass on a null test, where we know that the two sets of samples are equivalent. As our generative model, we use a Gaussian mixture model in $100$ dimensions, with $20$ components. We then repeat the following process $200$ times: We generate $5000$ samples from the Gaussian mixture model and then measure the $\chi^{2}_{\rm PQM}$ value of our test, with $n_{R}=100$. We show the results in Fig. 2. We can see that the $\chi^{2}_{\rm PQM}$ value is distributed as a chi-squared distribution with $n_{R}-1$ degrees of freedom, as expected from the discussion of § 2.2.

This test, while simple, is important, as it shows that the proposed method is not biased towards rejecting the null hypothesis. More importantly, it shows our main result introduced in § 2.2 in action. We show more such null tests for complex distributions in § A.

### 4.2 Gaussian Mixture Model

Figure: Figure 3: Gaussian Mixture Model. We show the results for PQMass on the Y axis, and the number of dropped modes on the X axis. We repeat the comparison $20$ times, each time changing the reference points, and report the standard deviation. We can see that the $\chi^{2}_{\rm PQM}$ value increases as we drop more modes, as expected.
Refer to caption: x3.png

For our second test, we re-use the Gaussian Mixture Model from § 4.1. We use our test, dropping some of the modes from the Gaussian Mixture Model. For each number of modes, we first generate $5000$ samples and then repeat our test $20$ times with $n_{R}=100$, changing the random reference points each time. We show the results in Fig. 3. We can see that the $\chi^{2}_{\rm PQM}$ value increases as we drop more modes, as expected. This shows that PQMass can detect when a generative model is missing modes and, therefore, has poor fidelity and diversity.

### 4.3 Validation of Sampling Methods

**Table 1: Comparison of various sampling methods on a mixture model in $D=2$ and a funnel in $D=10$. We show the results for PQMass and compare them to the Wasserstein distance, the MMD with an RBF kernel, and the Jensen-Shannon divergence.**
|  | Mixture Model ($D=2$) | Funnel ($D=10$) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | $\chi^{2}_{{\rm PQM}}$ ($n_{R}=100$) | $\mathcal{W}_{2}$ | RBF MMD | JSD | $\chi^{2}_{{\rm PQM}}$ ($n_{R}=100$) | $\mathcal{W}_{2}$ | RBF MMD |
| MCMC | $1621.06\pm 82.40$ | $8.94$ | $0.0151$ | $0.42$ | $855.40\pm 49.60$ | $17.41$ | $0.0566$ |
| FAB | $678.22\pm 88.43$ | $5.77$ | $0.0038$ | $0.39$ | $96.69\pm 13.52$ | $22.08$ | $0.0032$ |
| GGNS | $109.32\pm 12.36$ | $3.2$ | $0.0011$ | $0.35$ | $472.99\pm 18.00$ | $32.20$ | $0.0358$ |

Figure: Figure 4: Differences in various sample-based metrics as a function of the dimensionality of the data. We use a mixture of $10$ equally weighted Gaussians with varying numbers of dimensions. For each dimensionality, calculate the value of both metrics when comparing samples from the same distribution (blue) and from different distributions where one of them is missing one mode (orange), repeating the test various times to get an estimate of the noise. Ideally, we want our sample-based metrics to be able to detect that a mode has been dropped.
Refer to caption: x4.png

Figure: Figure 5: Computational cost of various sample-based metrics, as a function of the dimensionality of the data. For the same experiment as in Fig. 4, we show the time required to calculate the value of each metric, both as a function of the dimensionality of the data (left) and the number of samples (right).
Refer to caption: x5.png

In this section, we show how we can use PQMass to study the performance of sampling algorithms as long as we have access to true samples from the probability distribution. Sampling methods are often evaluated on synthetic tasks, such as sampling from a Gaussian mixture model, for which we can generate true samples. Sample-based metrics that are commonly used to evaluate sampling methods include the Wasserstein distance, the Sinkhorn distance (Cuturi, ), the Kulback-Leibler divergence (Kullback & Leibler, ), the Jensen-Shannon divergence (Lin, ), among others. While these sample-based metrics work well on low-dimensional settings, they are not well suited for high-dimensional settings, as they require the calculation of integrals over the space of samples, or a density estimation step, which is not feasible in high dimensions.

On the other hand, the proposed method is only based on calculating distances between points and can, therefore, be easily applied to high-dimensional settings. Furthermore, the method does not make any assumptions about the underlying distribution and, therefore, can be applied to any distribution as long as we have access to true samples.

First, we compare PQMass to other sample-based metrics on a low-dimensional task, where we can use other traditionally used sample-based metrics. We use the two-dimensional Gaussian mixture model used in Midgley et al. (), which has been used for benchmarking various sampling methods. We generate samples using Flow annealed importance sampling bootstrap (FAB, Midgley et al., )
Markov Chain Monte Carlo (Metropolis et al., ), for which we use the emcee implementation (Foreman-Mackey et al., ), and Gradient-Guided Nested Sampling (GGNS, Skilling, ; Lemos et al., ). For each method, we generate $10,000$ samples and compare them with $10,000$ samples from the true distribution. We calculate the Wasserstein distance, the Mean Maximum Discrepancy (MMD, Gretton et al., ) with a radial basis function (RBF) kernel and the Jensen-Shannon divergence (after performing a step of kernel density estimation); and compare them to the $\chi_{\rm PQM}^{2}$ value of our test, with $n_{R}=100$. For PQMass, we repeat this process $20$ times, each time changing the reference points, and report the standard deviation. We show the results in Table 1. We can see that PQMass is in good agreement with the other sample-based metrics and that the $\chi^{2}_{\rm PQM}$ value is correlated with the other sample-based metrics.

We then repeat this experiment on a higher dimensional task: Neal’s funnel distribution (Neal, ), a 10-dimensional distribution that is commonly used to benchmark sampling methods. We repeat the same experiment as before, except we do not use the Jensen-Shannon divergence, as it requires a density estimation step, which is not feasible in high dimensions. We see that PQMass is in good agreement with the MMD, while the Wasserstein distance follows a different pattern. We show samples from each model in § D, which show that, visually, PQMass captures the best-performing sampling methods.

There are, of course, multiple other methods for sampling, both deep learning based (such as Zhang & Chen, ; Lahlou et al., ; Richter et al., , amongst many others) and non-deep learning based (e.g., Del Moral et al., ; Hoffman et al., ), that could have been added to Table 1. Furthermore, hyperparameter tuning could be used to improve the performance of each sampling method. The purpose of this table is not to compare different sampling methods but rather to show that PQMass can be used to evaluate sampling methods.

Finally, we study the scaling of PQMass with the dimensionality of the data. We use a mixture of $10$ equally weighted Gaussians with varying numbers of dimensions. For each dimensionality, we generate $5,000$ samples and compare them with $5,000$ samples from the same distribution. We calculate the $\chi^{2}_{\rm PQM}$ value of our test, with $n_{R}=100$, as well as other sample-based metrics. We then repeat the experiment, dropping one mode and comparing the value of each metric. We do not compute the Kulback-Leibler or Jensen-Shannon divergence, as they require a density estimation step, which is not feasible in high dimensions. We show the results in Fig. 4. We can see that while $\mathcal{W}_{2}$ and linear MMD get increasingly noisier as the dimensionality increases, PQMass remains stable. MMD with an RBF kernel can also consistently highlight the difference between the two, however the computational cost of radial basis MMD scales far worse with dimensionality than PQMass. We show this in Fig. 5. We can see that the computational cost of PQMass scales much better with both dimensionality and number of samples than the Wasserstein and the MMD with an RBF kernel. MMD with a linear kernel is the cheapest computationally, but as shown in Fig. 4, it is too noisy to be usable in complex problems.

### 4.4 Time Series

Figure: Figure 6: Time Series. We show the results for PQMass on the Y-axis and the amplitude of the signal on the X-axis. We repeat the comparison $5000$ times, each time generating a time series with $A=0$, and a time series with $A\neq 0$. We can see that the $\chi^{2}_{\rm PQM}$ value increases as $A$ grows, as expected. We also show the values of $\chi^{2}_{\rm PQM}$ corresponding to the $3\sigma$ and $5\sigma$ significance levels of detection as black dashed lines.
Refer to caption: x6.png

Figure: Figure 7: Examples of time series with $A=0$ (left), and with $A\approx 0.12$ (right). PQMass can detect the signal with $5\sigma$ significance, a signal that is invisible to the naked eye (at least, to those of the authors).
Refer to caption: x7.png

For our next experiment, we show the flexibility of PQMass by applying it to a very different type of dataset: Time series data. We design the following experiment: We observe a noisy time series of fixed length, and we want to know if there is an underlying signal hiding in the noise. We generate the time series as follows: We generate the signal as:

$$ $y(t)=A\cdot\cos(t)+\eta(t),$ (18) $$

where $A$ is the amplitude of the signal and $\eta(t)$ is i.i.d. Gaussian noise with zero mean and unit variance. If $A=0$, then the time series is just noise. If $A\neq 0$, then there is a signal hiding in the noise. For each observation, we generate $100$ datapoints between $t=0$ and $t=10$. We then repeat the following process $5000$ times: We generate a time series with $A=0$ and a time series with $A\neq 0$. We then compare the two time series using PQMass, with $n_{R}=100$. We show the results in Fig. 6 for varying values of $A$. We can see that the $\chi^{2}_{\rm PQM}$ value increases as $A$ grows, as expected. The plot also the values of $\chi^{2}_{\rm PQM}$ corresponding to the $3\sigma$ and $5\sigma$ significance levels of detection. We see that, for this experiment, we can detect the signal with $5\sigma$ significance for $A\approx 0.12$, a signal that is invisible to the naked eye. We show an example of a time series with this amplitude, compared to one without signal in Fig. 7.

This experiment serves to show the versatility of PQMass. Because we make no assumptions about the underlying distribution, we can apply PQMass to any type of data as long as we have access to samples. Detecting signals in noisy time series is a common problem in multiple disciplines, such as astronomy (Zackay et al., ; Aigrain & Foreman-Mackey, ), finance (Chan, ; Sezer et al., ), and anomaly detection (Ren et al., ; Shaukat et al., ). Existing methods rely on assumptions about the underlying distribution. PQMass, on the other hand, can detect that the observed signal is not consistent with samples of random noise with no assumptions on the generative process. We leave the application of PQMass to these domains for future work.

### 4.5 Training a Generative Model

In this section, we track the value of our metric as we train two generative models, a variational autoencoder (VAE; Kingma & Welling, ) and a denoising diffusion model (Ho et al., ; Song et al., ) with hyperparameters detailed in § E. We train on the MNIST(^2^22[http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/)) dataset and compare generated samples with the MNIST test set. After each epoch of training is completed, we generate $2000$ samples and compute $\chi^{2}_{PMQ}$ with $n_{R}=100$. We execute this experiment over the first $100$ epochs to highlight the early stages of training for the models and repeat the experiment twice for robustness. We see in Fig. 8 that as training continues, the metric begins to stabilize with fluctuations due to the unique samples generated at each epoch. For the VAEs, we see that the samples can capture high-level structures, but they struggle to capture the complex features. Whereas the diffusion models, the generated samples are similar to the true distribution as seen by their low $\chi^{2}_{PQM}$ values, which plateau at around $\chi^{2}_{PQM}\sim 99$ as expected.

Figure: Figure 8: After each training epoch for the different models is completed, we generate samples and compute the $\chi^{2}$ metric to assess the quality against the true distribution. We perform this experiment twice, Run 1 and Run 2 for each model respectively, and see $\chi^{2}_{PMQ}$ values agree with each other even as they generate unique samples each time. The diffusion model can achieve $\chi^{2}_{PMQ}$ near ideal ($\chi^{2}_{PMQ}$ equal to the number of degrees of freedom $n_{R}-1$, in this case $99$) , indicating its ability to learn the underlying distribution.
Refer to caption: x8.png

The use of the metric for the generative model gives the ability to track the quality of the model’s performance relative to the true distribution, preserving the integrity of the underlying data structure and without any loss of dimensionality. Therefore, given samples from the true distribution (i.e. a test set), this approach can be used as a method to indicate performance throughout the training of a generative model.

### 4.6 Image Datasets

Figure: Figure 9: Correlation of human error rate in identifying real images with various sample-based metrics (results of prior methods taken from Jiralerspong et al. ()). The result for PQMass is the $\chi^{2}_{\rm PQM}$ value averaged over 20 samples. PQMass estimates the fidelity and diversity of generative models well despite not relying on a feature representation, unlike FLS and FID.
Refer to caption: x9.png

In this section, we test PQMass on image datasets. We use samples from Stein et al. (), which contains samples from various image datasets, and various generative models. We focus on the CIFAR-10 dataset(^3^33[https://www.cs.toronto.edu/~kriz/cifar.html](https://www.cs.toronto.edu/%C2%A0kriz/cifar.html)).
Following (Stein et al., ), we first use the human error rate as a measure of the fidelity and diversity of the generative models. We compare generated samples from each of the models, to the test data for CIFAR-10. We repeat the comparison $20$ times, each time varying the reference points. We show the results for CIFAR-10 in Fig. 9. We find that our chi-squared values strongly correlate with the human error rate, indicating that PQMass can measure the fidelity and diversity of generative models.

One key strength of PQMass is its scaling to high-dimensional problems. Unlike alternative methods such as FID and FLS, PQMass does not require using a feature extractor to reduce the dimensionality of the problem. Therefore, we can use PQMass to compare generative models on high-dimensional problems and do not risk potential biases introduced by the feature extractor.

### 4.7 Novelty

Figure: Figure 10: Novelty. PQMass (Y axis) when a fraction of the generated samples are replaced by samples taken from the training set (X axis). We repeat the comparison $20$ times, each time changing the reference points, and report the standard deviation. We show the value of $\chi^{2}_{\rm PQM}$ in the left panel, for the test (red) and train (blue), and the p-value for the training data on the left. We see that, as memorization increases, the gap between the train and test $\chi^{2}_{\rm PQM}$ increases and the p-value goes down.
Refer to caption: x10.png

One important aspect of generative models is their novelty: We want to know if the generative model generates samples that are novel or if it simply copies the training data. While PQMass can measure the fidelity and diversity of generative models, it cannot directly measure their novelty. There are two approaches we can follow to detect memorization: The first one is to follow Jiralerspong et al. (), in looking at the generalization gap for PQMass, i.e., how our metric changes when comparing the generated samples to the training data and when comparing the generated samples to the validation data. The second one is to look at the value of $\chi^{2}_{\rm PQM}$ on the training data directly. Indeed, we know that a value that is too low is indicative of overfitting. For a large number of regions (i.e., degrees of freedom), we know the chi-square distribution is approximately Gaussian. Therefore, we can use the symmetry of this distribution (as shown in Fig. 2, to get a p-value that penalizes suspiciously low values of $\chi^{2}_{\rm PQM}$ by considering the mirror reflection of the chi-square value around the maximum:

$$ $\text{p-value}_{{\rm overfit}}(\chi^{2}_{\rm{PQM}})\equiv\int_{-\infty}^{2n_{R }-\chi^{2}_{\rm PQM}}\chi^{2}_{n_{R}-1}(z)dz.$ (19) $$

To study this, we repeat the ‘copycat’ experiment of (Jiralerspong et al., ): We pick one of the models used in § 4.6, and repeat our test, replacing some fraction of the generated samples with samples from the training data. We then calculate the $\chi^{2}_{\rm PQM}$ value of our test, first comparing the generated samples to the training data and then comparing the generated samples to the validation data.

We show the results for PFGMPP in Fig. 10. The left panel shows that the gap between $\chi^{2}_{\rm PQM}$ when compared to the training data and $\chi^{2}_{\rm PQM}$ value when comparing to the validation data is larger the more samples we replace. The right panel shows the two-sided p-value, which clearly goes down as the number of copied samples increases. This shows that both of these methods (generalization gap and two-sided p-value) can be used to detect memorization in generative models.

## 5 Conclusions

In this paper, we have introduced a new method for quantifying the probability that two sets of samples are drawn from the same probability distribution. PQMass is based on comparing the probability mass of the two sets of samples in a set of non-overlapping regions. It relies only on the calculation of distances between points and, therefore, can be applied to high-dimensional problems. It does not require training or fitting any models. Furthermore, it does not require any assumptions about the underlying distribution and, therefore, can be applied to any type of data as long as we have access to true samples. We have shown that PQMass can be used to evaluate sampling methods and track the performance of generative models as they train.

We have shown the performance of PQMass on a variety of synthetic tasks, as well as on comparing sampling methods, comparing generative models of images, and detecting hidden signals in time series. Given the versatility and low computational cost of PQMass, it can serve as a valuable tool for evaluating the quality and performance of generative models and sampling methods.

## Acknowledgments

This research was made possible by a generous donation by Eric and Wendy Schmidt with the recommendation of the Schmidt Futures Foundation. This work is also supported by the Simons Collaboration on ”Learning the Universe”. The work is in part supported by computational resources provided by Calcul Quebec and the Digital Research Alliance of Canada. Y.H. and L.P. acknowledge support from the Canada Research Chairs Program, the National Sciences and Engineering Council of Canada through grants RGPIN-2020- 05073 and 05102, and the Fonds de recherche du Québec through grants 2022-NC-301305 and 300397. N.M. acknowledges funding from CIFAR, Genentech, Samsung, and IBM.

We thank Adam Coogan, Sebastian Wagner-Carena, Joey Bose, Gauthier Gidel, Alex Tong, Alexandra Volokhova and Marco Jiralespong for very inspiring discussions, and feedback on early versions of this work. We also thank Marco Jiralespong for sharing useful data.

## Reproducibility Statement

An implementation of PQMass, along with notebooks to reproduce the results from the experiments will be released upon publication.

## Impact Statement

As a robust and efficient algorithm for measuring discrepancies between sets of samples in high dimensions, PQMass can be used in the assessment of ML models as well as in scientific or industrial applications. The ability of PQMass to distinguish machine-generated and naturally occurring data distributions has implications for the detection of synthetic content, which is especially important in light of the growing emphasis on the interaction of AI systems with intellectual property, human rights, and social welfare.

## Appendix A Null Tests

Figure: Figure 11: Null tests, similar to the one shown in Fig. 2, but for two different distributions: Samples from MNIST (left) and a VAE trained on MNIST (right). The main result of our paper holds for samples from any distribution as long as we are comparing two sets of independent samples from the same distribution. Note that, on MNIST, the fit is not perfect due to the limited number of data points.
Refer to caption: x11.png

The main result of this paper, shown in § 2.2, was shown in practice in § 4.1 for a high-dimensional mixture of Gaussians. In this section, we show that it works in even more complex and high-dimensional distributions. First, we test it on the MNIST data: Using the MNIST observations themselves as samples from some underlying probability distribution. We put together the MNIST train and test data, leading to $70,000$ images. We then split them into two subsets of $35,000$ images each. To repeat the null test of § 4.1, we need independent samples at every iteration (note that this is different from the error bars reported in the rest of the experiments, which just arise from repeating with random tesselations, but always the same samples). Therefore, for each iteration, we take $1,000$ images from each dataset, and perform the PQM test with $n_{R}=100$, leading to 35 data points. We plot their histogram in the left panel of Fig. 11. While noisy, due to the very few data points, we see that the histogram does follow a chi-squared distribution with $n_{R}-1$ degrees of freedom, as expected.

For our second experiment, we use one of the MNIST generative models described in § 4.5. In this case, we can generate as many samples as we want. Because we are comparing the generative model to itself, the similarity of the generated samples to the MNIST train or test data is irrelevant for this particular test.

## Appendix B Bayesian Experiment

Figure: Figure 12: Log-probability of various models (translationally shifted by $l$) given a test dataset from a ground truth distribution. This probability is calculated using the Bayesian method described in Eq. 15. The ground truth distribution corresponds to $l=0$.
Refer to caption: x13.png

In this experiment, we numerically test the validity of the Bayesian method described in Eq. 15. This involved integrating the product of two multinomial distributions to determine the probability of a model given the training data. The setup mirrors that of the previous Gaussian Mixture Model experiment: we created a ’ground-truth’ distribution, a Gaussian Mixture Model in 100 dimensions with 20 components. The means and variances of these components were randomly selected. From this mixture, we drew 5000 samples to form the test set.

We then introduced a shift to this distribution along the diagonal direction (characterized by a vector of ones in 100 dimensions) multiplied by a scalar magnitude $l$. The resulting models, parameterized by $l$, were considered as potential generative models, aligning with the ground truth when $l=0$. Note that this is an arbitrary choice to construct a family of competing models.

Three non-overlapping regions were then defined by selecting three random points and constructing a Voronoi tessellation. We formed a regularly spaced grid over the values of
$\lambda_{1}$ and $\lambda_{2}$ (representing the probability mass in the first two regions), ensuring that $\lambda_{3}$ was chosen to allow the combination of the three values to be appropriately normalized.

We drew samples for each model (defined by different $l$ values on a regularly spaced grid) and calculated the model’s probability with Eq. 15. This was achieved by computing the appropriately normalized multinomial distributions for the test data and the model data within the three regions and then integrating their product. This integration was done by summing across all grid points in the probability mass and multiplying by the volume of each grid point’s probability.

Fig. 12 shows the results: the left panel illustrates the log probability, while the right panel presents the (un-normalized) probability of the models as a function of the single parameter $l$. As anticipated, the correct model ($l=0$) has a high probability, with the probability dropping as $l$ diverges from this value. Employing a greater number of smaller regions could enhance the constraining power; however, this would require a sample-based integration method over the unknown $\lambda$ values.

## Appendix C Ablation Study

**Table 2: Ablation study for CIFAR-10. We repeat the experiment of § 4.6, varying the number of reference points, as well as changing the distance metric to L1. We show the results for the frequentist version of PQMass, sorted in order of decreasing $\chi^{2}_{\rm PQM}$ for the default version. We find that the results are robust to the choice of the number of reference points and the distance metric. Any observed differences are within the standard deviation of the experiment.**
| Model | Default | $n_{R}=50$ | $n_{R}=200$ | L1 Distance | Inception |
| --- | --- | --- | --- | --- | --- |
| ACGAN-Mod | $12407\pm 10$ | $10410\pm 808$ | $14125\pm 995$ | $12336\pm 944$ | $7891\pm 1009$ |
| LOGAN | $713\pm 72$ | $408\pm 69$ | $1180\pm 95$ | $728\pm 108$ | $5750\pm 535$ |
| BigGAN-Deep | $268\pm 30$ | $135\pm 16$ | $498\pm 36$ | $238\pm 32$ | $615\pm 74$ |
| iDDPM-DDIM | $238\pm 22$ | $143\pm 17$ | $470\pm 41$ | $244\pm 31$ | $533\pm 78$ |
| MHGAN | $234\pm 39$ | $123\pm 19$ | $470\pm 48$ | $194\pm 18$ | $628\pm 64$ |
| StyleGAN-XL | $230\pm 13$ | $119\pm 15$ | $477\pm 23$ | $203\pm 23$ | $199\pm 19$ |
| StyleGAN2-ada | $207\pm 26$ | $128\pm 25$ | $417\pm 48$ | $197\pm 23$ | $259\pm 30$ |
| PFGMPP | $177\pm 22$ | $108\pm 18$ | $335\pm 28$ | $175\pm 16$ | $239\pm 28$ |

In this section, we study the effect of varying the two hyperparameters of our experiment: The number of reference points $n_{R}$, and the distance metric (which in all experiments in the main text is L2). We repeat the experiment on CIFAR10 described in § 4.6, varying the number of reference points, as well as changing the distance metric to L1. We also show the results when using the Inception-v3 feature extractor (Szegedy et al., ), as done in previous work such as (Jiralerspong et al., ).
We find that the order of the models is robust to the choice of the number of reference points and the distance metric, as well as to the use of a feature extractor. Any observed differences are within the standard deviation of the experiment. We show the results in Table 2.

## Appendix D Samples for Validation

Figure: Figure 13: Samples from our the various sampling methods introduced in § 4.3, and their corresponding PQMass values.
Refer to caption: x14.png

In § 4.3, we showed the performance of PQMass, comparing samples from various sampling algorithms to true distribution samples. Fig. 13 shows the samples from each algorithm and the underlying distribution. For the Gaussian Mixture Model (top), we see that MCMC is missing a mode, while in FAB the samples are too noisy; for Neal’s funnel (bottom), we see that FAB’s samples look indistinguishable from true samples, while GGNS look similar but more spread out, and GFN and MCMC fail to model this distribution. In both cases, the results from $\chi^{s}_{\rm PQM}$ correlate well with the similarity to true samples we can see by eye.

## Appendix E Generative Model Hyperparameters

In this section, we detail the hyper-parameters used for the respective models in  § 4.5. We first note that we use the MNIST data with the channel set to 1, since it is grayscale, and the spatial dimensions are $28\times 28$. For the variational autoencoder (VAE), the batch size is defined to be 512. For the encoder, the first layer maps from 784 dimensions to 512, followed by a second layer, reducing it further to 256 dimensions. The latent space was represented with 20 dimensions, as determined by the output of the subsequent layers. The Decoder architecture was the same, progressively reconstructing the data from the 20-dimensional latent space back to the original 784-dimensional space. We utilize the Adam optimizer with the learning rate initially set to 0.001. We also include a scheduler to reduce the learning rate every 50 steps by 0.1. For the denoising diffusion model, we use the package score_models(^4^44[https://github.com/AlexandreAdam/torch_score_models](https://github.com/AlexandreAdam/torch_score_models)). Our model utilizes the NCSN++ architecture with Variance Preserving Stochastic Differential Equation (SDE). The batch size is defined to be 256 with a learning rate of 0.001 and exponential moving average decay of 0.999.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: I would like to hear authors' opinions about the detailed weaknesses of their proposal.
soundness: 3
strengths: - PQMass introduces a new, likelihood-free approach for evaluating generative models by comparing sample distributions without density estimation or feature extraction, which is a fresh alternative to traditional metrics like FID and MMD.

- The experiments are comprehensive, testing PQMass across various generative models, data types, and dimensions. The paper includes comparisons with established metrics, scalability tests, and ablation studies, which aims to show PQMass’s robustness, scalability, and ability to assess fidelity, diversity, and novelty effectively.

- PQMass offers a potentially practical, scalable solution for evaluating fidelity, diversity, and novelty in generative models across diverse, high-dimensional data domains, filling relevant gaps in current model evaluation techniques.
summary: The paper introduces PQMass, a likelihood-free approach for assessing the quality of generative models by comparing sample distributions without estimating probability densities. PQMass operates by dividing the sample space into non-overlapping regions, using chi-squared tests to measure the similarity between the distributions of real and generated samples in these regions. The method yields a p-value indicating whether samples from two sets arise from the same distribution. Authors argue that PQMass offers several advantages: it does not require auxiliary models or feature extraction, works across various data types, and scales to moderately high-dimensional data. Through experiments, authors aim to show that PQMass is effective in evaluating fidelity, diversity, and novelty in generated samples, providing a robust alternative for assessing generative models in machine learning.
weaknesses: - The use of L2 or L1 distance metrics may limit PQMass's effectiveness, especially when the data resides on a complex manifold. These metrics may not capture meaningful differences in such cases, potentially leading to inaccurate assessments. 

- Experimental evaluation mostly focus on synthetic data or standard datasets like MNIST and CIFAR-10. Testing on more complex, real-world datasets would strengthen PQMass’s claims about the performance of the proposal.

- Although the paper briefly addresses other data types, it primarily evaluates PQMass on image data. Expanding experiments to more complex, structured data (e.g., text or biological data) and analyzing the impact of different distance metrics in these contexts would help to better validate PQMass performance.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: The paper is well presented and understandable. A broad range of examples and simulations were presented demonstrating fairly broad applicability. The test statistic is a basic and well known statistical quantity, so the method is fast to implement.
summary: The paper proposes a method for testing for the equality of two distributions from which two samples are obtained, by partitioning the sample space and comparing the multinomial vectors of counts of samples in each of the partitions via a chi-squared-distributed test statistic. The approach works for broad classes of data.

EDIT: Following the review/discussion period and (prospective) changes to the manuscript by the authors, I am increasing my score by 1.
weaknesses: Not clear if this is a weakness or a strength, but the proposed test is a well known, basic statistical test. The contribution of this paper, therefore, is to note that its easy to apply this test to a broad range of data types in order to detect a difference between the generating distribution of two samples. It's not clear if this is substantial enough a contribution for ICLR, though I do not wish to imply that substantial contributions need to involve complex methods.

Consistency Guarantees:
I'm not quite clear on or convinced about some of the claims being made.
* a) Proposition 1: Does the consistency here require any conditions on m and n as they approach -> infinity? E.g., what about their relative convergence rates (does m->infinity faster than n, for example). E.g. what if m/n -> K, where K=infinity, a finite constant, or zero. Presumably K=infinity or K=0 would break things and the test would no longer work (especially as you're sampling finite n_R samples).
* b) Proposition 1: Some clarification on the claim also needed here, as it's conditional on the n_R samples. E.g. if p = t_{\nu_p} and q = t_{\nu_q} (univariate t distributions with different degrees of freedom), and z_1 = -z_2, then you will never distinguish p and q under any rates of m, n -> infinity.
* c) Proposition 2: This proposition is ok, but really this situation won't happen, as the method explicitly requires that the z_i are taken from the set of (x_i, y_j), and so n_R is bound by m+n which are fixed and finite in this proposition (though the method will fail far before this as there will be no samples left to calculate the test statistic), as noted in the discussion. So how useful is Prop 2 in practice, particularly for small m+n?
* d) Proposition 2: "no information is lost by the PQMass statistic." would seem to be a bit of an overextension in light of the above point. But also because for n_R->\infty, the distribution of the statistic under the null would be a chi-squared distribution with infinite degrees of freedom. "No information is lost" only happens in this limit (as otherwise with a finite n_R, information can clearly always be lost), but how does your test statistic compare to its null distribution in this limit?

Practical implementation comments:
* a) As far as I can tell, you've only considered exactly balanced samples each time: m=n. While in some applications the user has control over both m and n, this will not be the case in many scenarios. So how well does the test perform when the two samples are imbalanced, including extreme cases?
* b) Using reference samples z_i randomly taken from the two distributions (x_i) and (y_i) has a long history, even when comparing samples from two MCMC samplers (or the same MCMC sampler twice, when considering convergence diagnostics; e.g. Sisson & Fan (2007). Statistics and Computing, 17, 357-367). So this approach is fine. However, a disadvantage of this is that when p and q differ only, say, by a small mode with low weight in the tail, then by sampling uniformly over the x_i and y_j, one is reasonably likely to miss samples in this mode, as the sampling will naturally favour repeated samples in areas of higher density, and hence the test will miss the obvious difference between p and q. (The examples in Section 3 seemed to consider equally weighted mixtures only ...). Hence the need to repeat the test multiple times with different z_i. Is there any argument to be made to reconsider this equal sampling of the z_i from (x_i) and (y_j), to ensure there is a greater spread over the space, rather than doubling (and tripling, and ...) down on high density regions?
* c) Figure 2: presumably when "this process" is repeated 500 (or 200?) times, this also means resampling the z_i? (Could be clearer in the caption.) But really, this histogram doesn't really seem to follow the red density. Suggest that the number of reps is increased here considerably if you're wanting to claim these distributions are the same. Also the y-axis label of "Frequency" seems to be incorrect, as the axis ticks aren't measured in counts.
* d) In the results in general, the empirical mean of the PQMass statistic (4) seems to be taken as "the statistic" for determining whether p=q (or p!=q). Is it obvious that this mean statistic has a chi-squared(n_R) distribution under the null? Each of the components of this statistic are chi-squared(n_R) (conditional on z_i) under the null, but is it obvious that they are independent from eachother? (chi-squared(m) + chi-squared(n) = chi-squared(m+n) if the two LHS distributions are independent). Certainly the alternative distributions are not the same for different z_i. So can this "mean PQMass statistic" actually be compared to the chi-squared null distribution in any meaningful way? Or is the use of the mean PQMass statistic merely limited to relative value comparisons? In which case, the paper should be clearer about all of this and tone down the chi-squared relevance.
* e) Section 4: Limitations. The method also appears to require independent samples (x_i), (y_i), otherwise the multinomial model does not hold. How likely is this in practice? In the MCMC comparisons in Table 1? Etc.


Typos:
* p.2 para 2, wrong parentheses surrounding reference list starting with Theis et al.
p.5. Algorithm step 1. "choice" -> "chosen"
p.6. Section 3.1. The number of test replicates here is 200, but in Figure 2 it's listed as 500. One of these must be wrong.
p.6. Table 1. D was previously the distance metric between two sample points (c.f. Section 2.2, Algorithm). Now D has switched to number of dimensions. Maybe separate the notation?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: What are the strengths or advantages of the proposed method compared to other distribution comparison techniques beyond MMD and W2?
soundness: 2
strengths: The strength of the proposed method is in its simplicity. After discretizing the data space, one only needs to count the data points in each region and calculate the test statistic. This makes the method computationally much more efficient compared to existing distribution comparison methods, such as MMD.
summary: The authors proposed a simple method for comparing the distribution $q$ of a generative model with the true data distribution $p$. The proposed method first discretizes the data space and then compare the proportion of data points in each region. Since this becomes a distribution comparison for a multinomial distribution, it can be tested using Pearson’s $\chi^2$ test. If the test indicates a difference between the two distributions, one can conclude that $p \neq q$, making this method a sufficient condition for identifying distribution differences. The authors demonstrated the effectiveness of the proposed method on synthetic data as well as datasets such as MNIST and CIFAR-10.
weaknesses: A weakness of this study is the limited comparison with other two-sample test methods. The authors simplified the problem by dividing the data space and reducing it to a multinomial distribution comparison. However, various other methods have been proposed for distribution comparison not only MMD and W2. For example, [Ref1] explores distribution comparison using classifiers and its application to GAN evaluation, while simpler approaches, like those using nearest neighbors, are also available [Ref2]. Although the simplicity of the proposed method is an advantage, comparing its performance with these existing distribution comparison methods would be essential for evaluating its practical utility.

* [Ref1] Revisiting Classifier Two-Sample Tests, ICLR 2017.
* [Ref2] Multivariate Two-Sample Tests Based on Nearest Neighbors, Journal of the American Statistical Association, 1986.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Methodological questions:

Is there any particular reason that you pursued Pearson’s chi squared test ahead of Fisher’s exact test, which should be less vulnerable to small contingency cell counts, which could easily happen here with the randomised voronoi tessellation?

Choice of centres:

Is it really hard to do better than random? Some sort of relatively simple diversity measure would potentially get you quite far here.
“Sample the reference points from a uniform mixture of the empirical distributions of samples from p and q”

What if there are more samples available from one distribution?

What if one of the distributions is known to be more accurate representation of reality and therefore give a better coverage of realistic regions of probability space?

“Perform the test multiple times”? How many is necessary? What are the computation implications?

“This effectively marginalizes the choice of tessellation”: this comment makes me wonder whether a Bayesian formulation of the problem would work well, since you are effectively having sample the latent allocation of the points to reference centres vs cell populations. There will be analytical conjugate Bayes solutions to the multinomial distribution comparison too, so there should be no more pesky posterior sampling in addition to what is being done already.

Data balance between distributions:

What if there are unbalanced numbers of samples available from each distribution? This will be quite common, since in general distributions are not equally easy to sample from. This is especially true if comparing simulated data and real data, or complex vs simpler models. How would this influence the sampling of reference points?

Limitations: good for you for being open “if the number of samples is too small”. Do you have any idea how small? Does it scale with p in a demonstrable way?
soundness: 3
strengths: The idea is pleasingly simple for a relevant contemporary challenge in stats/ML, and explained pretty clearly. The experimental results are strong and quite convincing, especially in higher-dimensional problems without an obvious feature representation.
The figure quality is high throughout, which is a good thing in general.
The experiments themselves are pretty thorough, as are the descriptions of the experiments, especially the astrophysics experiments that might be hard for a non-expert audience. The example detecting the small additive noise is nice in Appendix F.2. Demonstrating the method can find an effect invisible to the naked eye is good.
summary: The article proposes a new method for comparing samples from two distributions, providing a hypothesis test for whether they are identically distributed. The method introduces a Voronoi tessellation, dividing the data space into non-overlapping regions, and then assumes a multinomial likelihood describing the prevalence of observed data in each of the regions: this likelihood is then used to develop a hypothesis test as to whether each distribution has the same underlying probabilities of generating data in each region, which then implies equivalence of the distributions with some assumptions. Some theoretical results are derived to strengthen this argument. The method is then tested on several empirical problems.
weaknesses: The very high figure quality has the probably unintended consequence that the resulting pdf file size is very large (>17MB), which makes it kind of a pain to handle by conventional means. If there were some way to reduce this without overly compromising readability, then that would be good. I suspect that you have one or two figures in the appendix that are unnecessarily high resolution when printed out, but I haven’t figured out which ones (possibly figures 17/18).

A few small language issues:

“Miscrepancy” line 356. I assume you meant “discrepancy”? English in general pretty good.
“The instrument has multiple sources of non-Gaussian additive noise, including… Gaussian noise” is kind of odd.

Figure design:

Figure 6 is hard to understand immediately. The claim about “strongly correlate” (line 446) does not jump out intuitively from the data plotted.

Some of the descriptions of the method’s properties are a little overly flattering:

You call this a “Likelihood-free method”... but, you do use a multinomial distribution.
Desiderata 1: “PQMass works with the intervals of pdfs, it does not make assumptions about the true underlying density field”. The cdf is defined by the pdf, and assumptions about one will imply assumptions about the other, so I don’t think this argument is particularly strong.

“These results show that, at least for “well-behaved” densities, no information is lost by the PQMass statistics”
Did you not say at the start that you didn’t make assumptions about the underlying density?
“No information is lost” in what sense? Clearly information is lost when you quantise. Is this claim meant in some asymptotic limit?

More explanation of the 2-d Gaussian mixture model and the 10-d funnel distribution would be good rather than just a reference, if you can find the space. Appendix is always an option.

Meta-review: This paper suggests a method to assess how good a generative model is by a likelihood free probability mass estimation procedure. While I find some aspects heuristic, the reviewers are in agreement that there are good ideas here and their opinions improved after the rebuttal period. The method is appealing in its lean reliance on assumptions and its scalability.

Decision: Accept (Poster)