A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Science
Authors: Sebastian Bischoff
Alana Darcher
Michael Deistler
Richard Gao
Franziska Gerken
Manuel Gloeckler
Lisa Haxel
Jaivardhan Kapoor
Janne K Lappalainen
Jakob H. Macke
Guy Moss
Matthijs Pals
Felix C Pei
Rachel Rapp
A Erdem Sağtekin
Cornelius Schröder
Auguste Schulz
Zinovia Stefanidi
Shoji Toyota
Linda Ulmer
Julius Vetter
Venue: Accepted by TMLR ()

Abstract: Generative models are invaluable in many fields of science because of their ability to capture high-dimensional and complicated distributions, such as photo-realistic images, protein structures, and connectomes. How do we evaluate the samples these models generate? This work aims to provide an accessible entry point to understanding popular sample-based statistical distances, requiring only foundational knowledge in mathematics and statistics. We focus on four commonly used notions of statistical distances representing different methodologies: Using low-dimensional projections (Sliced-Wasserstein; SW), obtaining a distance using classifiers (Classifier Two-Sample Tests; C2ST), using embeddings through kernels (Maximum Mean Discrepancy; MMD), or neural networks (Fréchet Inception Distance; FID). We highlight the intuition behind each distance and explain their merits, scalability, complexity, and pitfalls. To demonstrate how these distances are used in practice, we evaluate generative models from different scientific domains, namely a model of decision-making and a model generating medical images. We showcase that distinct distances can give different results on similar data. Through this guide, we aim to help researchers to use, interpret, and evaluate statistical distances for generative models in science.

Published in Transactions on Machine Learning Research (08/2024)
A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Science
Sebastian Bischoff 1,2.3,4, Alana Darcher Michael Deistler Richard Gaol?, Franziska Gerken', Manuel Gloeckler Lisa Haxe|1.237 Jaivardhan Kapoor Janne Lappalainen Jakob H Macket.2 Guy Moss Matthijs Pals* 11.2, Felix Peil.2, Rachel Rapp* A Erdem Sağtekin'* 1 Cornelius Schröder! Auguste Schulz*1; Zinovia Stefanidi' Shoji Toyota", Linda Ulmer and Julius Vetter
'Machine Learning in Science, Excellence Cluster Machine Learning, University of Tübingen, Tübingen, Germany
TTbbigee AI Center, University of Tübingen, Tübingen, Germany
University Hospital Tübingen, University of Tübingen, Tübingen, Germany 4M3 Research Center, University Hospital Tübingen, Tübingen, Germany
UUniversity Medical Center of Bonn, Bonn, Germany
Dynnamic Vision and Learning Group, Technical University of Munich, Munich, Germany Hertic Institute for Clinical Brain Rescarch, Tübingen, Germany The Institute of Statistical Mathematics, Tokyo, Japant
Reviewed on OpenReview:
Abstract
Generative models are invaluable many fields of science because of their ability to capture high-dimensional and complicated distributions, such as photo-rcalistic images, protein structures, and connectomes. How do we evaluate the samples these models generate? This work aims to provide accessible entry point to understanding popular sample-based statistical distances, requiring only foundational knowledge in mathematics and statistics. We focus on four commonly used notions of statistical distances representing different methodologies: Using low-dimensional projections (Sliced-Wase Wasserstein; SW), obtaining a distance using classifiers (Classificr Two-Sample Tosts; C2ST), using embeddings through kernels (Maximum Mean Discrepancy; MMD), or neural networks (Fréchet Inception Dislance; FID). We highlight the intuition behind each distance and explain their merits, scalability complexity, and pitfalls. To demonstrate how these distances used in practice, evaluate generative models from different scientific domains, namely a model of decision-making and model generaling medical images. We showcase that distinct distances can give different results on similar data. Through this guide, we aim to help researchers to use, interpret, and ovaluate statistical distances for generative models in science.
1 Introduction
Generative models that produce samples of complex, high-dimensional data, have recently come to the forefront of public awareness due to their utility in a variety of scientific, clinical, engineering, and commercial domains (Bond-Taylor et al. 2021). Prominent examples include StableDiffusion (SD) and DALL-E for
generating photo-realistic images (Rombach et al. 2022a). WaveNet (Oord et al. 2016 for audio synthesis,
"Co-organizer
tCorresponding authors: manuel- www.:egggleeeeeetttteeeeeeeeeee matthijs.plls@ueeeeee sals@uni-tuubingen..d
:Current affiiiaion: Department Advanced Information Technology, Kyushn University, Fukuoka, Japan
Authors listed alphahetical order.

---

Published in Transactions on Machine Learning Research (08/2024)
and Generative Pre-trained Transformer (GPT: Radford et al 2018 2019: Brown et al. 2020 for text generation. In addition to the recent surge in generative models, many scientifie disciplines have a long history of developing data-generating models that capture specific processes. In neuroscience, for example, the occurrence of action potentials is commonly modeled at varying levels of detail (c.g., single neuron voltage dynamics; Hodgkin & Huxley 1952. or at a phenomenological level; Pillow et al. 2008); whereas in; e-g-, astrophysics there exist various models to simulate galaxy formation (Somerville & Davé 2015). Along with generating novel synthetic samples, generative models can be leveraged for specific tasks, such as sample generation conditioned on class labels (e.g., diseased vs. healthy brain scans Pombo et al. 2021 molecules that can or cannot be synthesized;; [Urbina et. al. 2022, class-conditional image generation; van den Oord et al. 2016 Dockhorn et al. (022), forecasting future states of a dynamical system (e.g., Jacobs et al. 22023] Bremner el al. 2022 Pals et al. 2024).
generating neural population activity conditioned on visual stimulii or 2018 Bashiri et al. 2021 Schulz et al. 2024 Kapoor et al. 2024 data
behavior (e.g.. Molano-
et al.
imputation (c.g.. Vetter ct al. 2023 Lugmayr et al. 2022). data augmentation for downstream tasks (Rommel
et al.
2022 and many more (see also Table [S]].
These powerful capabilities are enabled by the premise that generative models can produce samples from the high-dimensional distribution from which we assume our datasct was sampled. The dimensions can correspond to anything from individual pixels or graphs to arbitrary features of physical or abstract objects. When aiming to build generative models that better capture the true underlying data distribution, we need to answer a key question: How accurately do samples from our generative model mimic those from the true data distribution?
Manual inspection of generated samples can be a good first check, e.g., in image or audio generation, where can directly assess the visual likeness or sound quality of the samples (Gerhard et al. 2013 Vallez et al. 2022, Jayasumana et al. 2023). In general, however, we would like to quantitatively compare the similarity of distributions, for instance to benchmark different generative models. Many measures have been proposed that assess the similarity of two distributions based on various aspects of their moments or the overall probability density (Fig. 1. Some of these measures require likelihood evaluations, as is possible with generative models such as Gaussian Mixture Models, Normalizing Flows, Variational Autoencoders, autoregressive models or
diffusion models (Bishopp 2006 Papamakarios ct al.) 2021 Box et al. 2015 Kingma & Welling, 2014 Yenduri
et al. 2024 Ho et al.) 2020 Song et al.) 2021). However, many contemporary machine learning models (e.g.. Generative Adversarial Networks and Energy-Based Models; [Goodeelow et al. 2014 Rezende et al. 2014
Hinton et al. 2006) and scientific simulators (e.g,, single neuron voltage dynamics; Hodgkin & Huxley 1952
only define the likelihood implicitly, i.c., we can not explicitly evaluate their likelihood. Statistical distances that can be computed based on samples only are therefore invaluable for comparing both classes of generative models (and to real data) in scientific contexts.
Here, present guide that aims to serve an accessible entry point to understanding commonly used sample-based statistical distances. Towards this goal, we provide explanations, comparisons, and example applications of four commonly applicd distances: Blsser-Wassestecin (SW), Classifier Two-Sample Tests (C2ST), Maximun Mean Discrepancy (MMD), and Fréchet Inception Distance (FID). With these resources, we aim to empower researchers to choose, implement, and evaluate the usage and outcomes of statistical distances for generative models in science. Note that, with distance, we do not necessarily refer to a distance metric in the mathenatical sense (i.c., satisfying symmetry and the triangle inequality) but to a general measure of dissimilarity between two distributions (however, some considered distances are in fact metrics).
The general and didactic nature of this guide means it can neither be comprehensive nor provide a clear-cut answer as to which distance is 'best', as there are numerous choices, and the 'ideal' one strongly depends on the domain of application. Previous articles have provided useful and extensive reviews for specific use cases. For example, Theis et al.] (2016) discusses commonly used criteria for evaluating generative models of images, Borji (2119); Xu et al. (2018): ; Yang et al. (2023) compare metrics specific to evaluating GANs (including, e.g., specialized variants of the FID), Basseville (2013) provides an extensive overview of previous works on divergences, and Gibbs & Su (2002) analyze the theoretical relationships among 'classic' distances (including, e.g., Wasserstein and KL-divergence) Rather, by going through four different classes of sample-based distances in detail and systematically comparing them on synthetic and real-world applications,
2

---

Published in Transactions on Machine Learning Research (08/2024)
(x))(X)
Py())
P2(X)
p(x piX
00
X Yue )))
X, (0
X₂ (00)
dim1
D(X X,) > D(X x)
Figure The need for statistical distances in scientific generative modeling. (a) An example target distribution, (T), and two learned distributions (pi(z) and p2(2)) of different models trained capture Perue (z). All three distributions share the same mean and marginal variances, despite having distinct. shapes. However,
appropriate sample-based distribution distance D can determine that P2(x) is more similar to (z). (b) Scientific applications often require evaluating high-dimensional distributions, such as distributions of images or tabular data. In this example, each point represents an X-ray image, where each dimension is one pixel.
aim to provide a solid foundation for navigating the extensive literature on statistical distances, and to enable readers reason about other related distances not covered here.
The outline of this guide is as follows: First, we provide an intuitive and graphical explanation for each of the four distances (Section 2 We then perform a systematic evaluation of their robustness as function of dataset size, data dimensionality, and other factors, such as data multimodality (Section 3). Finally, we demonstrate how these distances can be applied to compare generative models different scientific domains (Soction We evaluate low dimensional models of docision making in behavioral neuroscience and generative models of medical X-ray images. We show the importance of using multiple complementary distances, distinct distances can give different results when comparing the same sets of samples.
2 Sample-based statistical distances
In this section, we provide an overview of lour classes of sample-based statistical distances commonly used in machine learning literature. Each class takes a different approach to overcoming the challenges inherent in comparing samples from high-dimensional and complex distributions. Throughout the section, we assume that we want to evaluate the distance between two datasets of samples, denoted as {ai, Fnx ~ p1 (x) and (y), .., }m p2 (y), where pı (xr) and p2(y) two probability distributions. These can be either two generative models, generative model and the underlying distribution of the observed data,
2.1 Slicing- based: Sirced-Wassersteii (SW) distance
Computing distance between distributions suffers from the curse of dimensionalliy, where the computational cost of computing the distance increases very rapidly as the dimensionality of the data increases. This problem is espocially restricting when the distance used as part of a loss function optimization problems, since in this case it must be evaluated many times. This has prompted the notion of "sliced" distances, which have become increasingly popular in recent years (Kolouri et al., 2019 Nadjahi et al. 2020 Goldfeld & Greenewald 2021). The main idea behind slicing is that for many existing statistical distances efficiently evaluate the distance in low-dimensional spaces, especially in single dimension. Therefore, the "slices" are lypically one-dimensional lines through the data space (Fig. 2a). All data points from each distribution are projected onto this line by finding their nearest point on the line, giving one-dimensional distribution of projected data points (Fig. pp). The distance mcasure of interest between the resulting one-dimensional distributions can then be computed efficiently. However, computing the random projection could lead to an unreliable measure of distance distinct distributions can produce the same one-dimensional
3

---

Published in Transactions on Machine Learning Research (08/2024)
a Slicing
1D projections
P
0
-1
Pa
Figure Schematic for the Sliced- Wasserstein distance. (a) Samples from two two-dimensional distributions along with example slices. The "slicing" is done by sampling random directions from the unit sphere and projecting the samples from the higher-dimensional distribution onto that direction. (b) One-dimensional projections of the two distributions correspouding to the two random slices in (a). For each pair of projections, the empirical Wasserstein distance is computed. Unlike in higher dimensions, this can be done efficiently for one-dimensional distributions.
projections. Therefore, we repeat the slicing process for many different slices and average the resulting distances. More formally, compute the expected distance in one dimension between the projections of the respective distributions onto (uniformly) random dirrctions the unit sphere. As long the distance of choice is a valid metric in one dimension, the sliced distance defined in this way is guaranteed to be valid metric as well (Nadjahi et al. 2020 Proposition (iii).. The most popular example of a sliced distance metric is the Wassersten (SW) distance (Fig. D The Wasserstein distance and its sliced variant have several attractive properties: They can be computed differentiably; their computations do not rely heavily on choices of hyperparameters; and the sliced variant very fast compute. However, we note that slicing has also been done for other distance measures, such MMD with specific choice of kernel (Hertrich et al 2024 and mutual information (Goldfeld & Greenewald 2021). We provide a formal definition of the Wasserstein distance below, and of the Wassssssssn distance in Appendix A.3
Definition of Wasserstein Distance We here provide the definition in the common case where probability density functions pı and P2 are assumed to exist (formal definition using probability measures in Appendix A.3 Let M C Rd, and ||- lle be the qonor in Rdd Then the Wasserstein-q norm can be written as
Wg(, inf (Fez, || x2 x2l1))
where P2) is the set of all couplings, that is all possible "transportation plans", between Pi and P2- 7 E (2) is a joint distribution over (21, with respective marginals Pı and p2 over 21 and xz-
Sample-Based Wasserstein Distance In practice, Eq. is analytically solvable for only a few distribu- tions. Therefore, Wasserstein distance typically estimated from finite samples from P1 and Pz. However, sample-based estimates of the Wasserstein distance are biased, and the convergence to the true Wasserstein distance is exponentially slower as the dimensionality of the distribution increases (Fournier & Guillin 2015 Papp & Sherlock 2022)
Intuitively, if two given probability distributions are thought of as two piles of dirl, the Wasserstein distance measures the (minimal) cost "transporting" one pile of dirt to another (Panaretos & Zemel 2019). This analogy led to the Wasserstein distance also being known as the Earth Mover's Distance. Suppose we have samples { C Ed sampled from a distribution Pı and {yi, y.. Rd sampled from another distribution p2. Given any distance metric between two vectors in Rdd D( we construct the cost.
4

---

Published in Transactions on Machine Learning Research (08/2024)
a
b
total cost 2.8
total cost = 1.9
cost
Figure 3: Computing Wasserstein distance. Two transport maps mapping the samples from a two- dimensional distribution pi (black) to samples from another distribution p2 (blue), shown by arrows. The color of the arrow corresponds to the cost (Euclidean distance) between z and ye. (a) Randomly chosen transport map. (b) The optimal transport map, giving the smallest total cost. The total cost for the optimal map in (b) is the Wasserstein distance belween these two sets of samples. Note that this schematic demonstrates transport maps for the non-sliced Wasserstein distance in two dimensions.
matriz C, as the matrix of pairwise distances between the samples 2, and yy:
D(11,81)
D(11.U))
C=
(2)
D(IN.y1) D(NN.yN))
Recalling the Earth Mover's Distance analogy, we want to map each x, to exactly one Vy. in such a way that the cost of doing so is minimized. The minimum transport map then defines the Wasscrstein distance (for the metric D) between the two empirical distributions. Throughout this work, we employ the commonly used Euclidean metric, L leading to the Wasserstein-2 and Sliced Wasserstein-2 distances. More precisely, we define a "transport map" be a permutation matrix, {0, N which a matrix with exactly one nonzero entry in each row. The entry Tyy means that we transport the point x; to the point y/- Then, finding the transport map that minimizes the overall cost be stated
min Σ Cuy.
A randomly chosen transport map for small datasets IR? is shown in Fig. 3a. Fortunately, the optimal solution to Eq. ( can be solved exactly using the Hungarian method (Kuhn 1955). leading the assignment
shown in Fig. 3
Slicing Wasserstein brings efficiency Solving the optimal transport problem (Eq. (3) with the Hun- garian method has time complexity of O(N3) in the number of samples N (although faster O(N2 log N) approximations exist, see Peyré et al. 2017). However, in the special case where the data is one- mne-dimeniional, the Wasserstein distance can be caleulated by sorting the two datascts, obtaining the order statistics {I(1),.., Z[N)} and (M(), ... } and computing the sum of the distances Σ, ((r M()). This has time complexity of O(N log N). Thus, slicing the Wasserstein distance with one-dimensional projections becomes very efficient. While the value of the SW distance does not converge to the Wasserstein distance, even in the case of an infinite number of data samples and slices, the SW distance is metric (in the mathematical sense) long D is metric on RRd, and it acts as lower bound to the Wasserstein distance (Nadjahi 2021
Limitations When two high-dimensional distributions differ only along a small subset of directions, relying on one dimensional projections can become a significant limitation. This is because in high-dimensional spaces, most slices will be (almost) orthogonal the few directions which distinguish the two distributions. In such situation, the gain in efficiency from slicing can be impeded by the large number of slices we would need to reliably tell the two distributions apart This limitation be addressed by considering
5

---

Published in Transactions on Machine Learning Research (08/2024)
dim2
oo
C2ST 0.5
C2ST = 0.7
C2ST 1.0
(,)
P2(x)
dim2
misclassifid
dim1
Figure 4: Classifier Two-Sample Test (C2ST). (a) The C2ST classifier problem: identifying the source distribution of a given sample. The optimal classifier predicts the higher-density distribution at every observed sample value, resulting in a majority of samples being correctly classified.
(b) When probability densities of the distributions are not known, the optimal classifier approximated by training a classifier, e-g., a neural network, to discriminate samples from the two distributions. (c) C2ST values vary from 0.5 when distributions exactly overlap (left) 1.0 when distributions are completely separable (right).
2018 slices. Furthermore, slicing may also be relaxed to other kinds of data-specifie projections, such Fourier features for stationary time series or locally-comnected projections for images (Du et al. 2023 Cazelles et al. 2021). Finally, while sliced distances can overcome computational challenges of computing distances in higher dimensions, they typically inherit the limitations of the distance being sliced.
nonlinear Kolouri et al. 2019) other specific (Deshpande et al.
2019
2.2 classifier-based Classifier Two-Sample Test (C2ST)
The Classifier Two-Sample Test (C2ST) uses classifier that discriminates between samples from two distributions (Fig. [a Lopez-Paz & Oquab 2017 Friedman 2003). The distance between the distributions can then be quantified by various measures of classifier performance. For example, one would train a classifier c(x) to distinguish samples from the generative model and the data, and then evaluate the C2ST as } (appp (c(x) = 0)] + (Fgt)) [1(c() 1)]]- The classification accuracy provides a particularly intuitive and interpretable measure of the similarity the distributions. the classification accuracy is 0.5, i.e., the classifier at chance level, the distributions are indistinguishable to the classifier (Fig- 4c, left), while higher accuracy indicate differences in the distributions (Fig. 4c. middle). If the C2ST is 1.0, the two distributions have no (or very little) overlap in their supports (Fig. c right). Given two distributions, the C2ST has 'true' (optimal) value, which is the maximun classification accuracy attainable by any classifier (Fig, 4 This optimal value can be computed if both distributions allow evaluating their densities, but this is not usually possible only data samples are available. In that case, one aims to train a classifier, such as a neural network (Fig. {)), that as close to the optimal classifier as possible.
One of the main benefits of the C2ST is that its value is highly interpretable (the accuracy of the classifier). C2ST also be used to test the statistical significance of the difference between two sets of samples. However, calculating C2ST can be resource-intensive because it involves training classifier. Additionally, using classifiers in differentiable training objective can be challenging (but is sometimes possible, as in, e.g., Generative Adversarial Networks; Goodfellow et al. 2014)). Furthermore, the value is dependent on the capacity of the classifier, and honce on many hyperparametors such as classifior architecture or training procedure. This dependence on a trained classifier can result in C2ST estimates that are biased, and the variety of possible classifier architectures means theoretical guarantees such as sample complexity are difficult. to determine. In our experiments, used scikit-learn Multi-Layer Perceptron classifier, combined with five-fold cross cross-validatinn validation routine to estimate the accuracy returned (Pedregosa et al. 2011
Common failure modes As mentioned above, for any realistic scenario the C2ST is computed by training classifier. The resulting C2ST will only be good measure of distance between real and generated data

---

Published in Transactions on Machine Learning Research (08/2024)
a
Fallure of C2ST to discriminate classes
b
C2STs for high-D data
c
High C2ST on MNIST
Data
Data
1.0
Ground truth Neural net
0.5
dim2
p(x)
Gaussian: C2ST=1.0
0.5
0.0
10
#samples
10
-3
X
3
Model
1.0
1.0
dim2
MoG: C2ST=1.0
C261
C257
0.5
0.5
dim1
16
10'
10
Hidden layer size
Dimensionality
Figure 5: Failure modes and behavior of C2ST. (a) Data (top left) and Gaussian maxximum-likelihoon estimate (bottom left). C2ST wrongly returns 0.5 (no difference between the densities) if too few samples are used (top right) or the neural network is poorly chosen (bottom right). (b) For high-dimensional densities, despite the marginals between data (black) and model (gray) seeming well-aligned, small differences (here a mean shift of 0.25 std. in every dimension) allow the classifier to more easily distinguish the distributions dimensionality increases, yiclding correct but surprisingly high C2ST. (c) On MNIST, the C2ST between data (top) and a Gaussian generalive model (middle) well as of Mixture of Gaussians (MoG, bottom) is 1.0, although the MoG is perceptually more aligned with the data.
if the classifier is close or equal to the optimal classifier. To demonstrate the behavior of the C2ST if this is not the case, we fitted a Gaussian distribution to data that was sampled from a Mixture of Gaussians (Fig. 5a, left). The optimal C2ST between these two distributions is 0.65 (which can be computed because Gaussians and Mixtures of Gaussians allow evaluating densities). Tf the C2ST estimated with a neural network, however, we observe that this C2ST can be systematically underestimated: for example, when only few samples from the data and generative model are available, the neural network predicts C2ST of 0.5 (Fig. p, top right)- in other words, it predicts that the generative model and the data follow the same distribution. Similarly, if the neural network is not expressive enough, c.g., with too few hidden units, the classifier will also return a low C2ST, around 0.5 (Fig, 55, bottom right). These issues can make the C2ST easy to misuse. In many cases, reporting a low C2ST desirable for generative models since it indicates that the model perfectly matches data, but one can achieve low C2ST simply by not investing sufficient time into obtaining strong classifier.
C2ST can remain very high even for scemingly good generative models We previously argued that the C2ST is an interpretable measure while this is generally true, the C2ST can sometimes be surprisingly high even if the generative model seems well aligned with the data. For example, when the generative model aligns very well with the data for every marginal, the C2ST can still be high the data is high-dimensional (Fig. 5)). Because of this, can be difficult to achieve low C2ST values on higl-dimensional data. To further demonstrate this, we fitted Gaussian distribution and a mixture of 20 Gaussian distributions to the 'ones' of the MNIST dataset. Although the Mixture of Gaussians (Fig. 5 bottom row) looks better than a single Gaussian (Fig. 5c. middle row), both densities have C2ST of 1.0 to the data (obtained with ResNet 244k held-out test. datapoints).
Other C2ST variants While we focus on a slandard C2ST definition by using classification accuracy as the C2ST distance [Lopez-Paz & Oquab 2017). any other performance measure for binary classification could used (Raschka) 2014). Kim et al. (2019) argues that classic accuracy sub-optimal due to the "binarization" of the class probabilities and proceeds to instead use the mean squared error between the predicted and "target" value of 0.5. Other approaches instead construct likelihood ratio statistic (Pandeva et. al. 2024
7

---

Published in Transactions on Machine Learning Research (08/2024)
Additionally, instead of using the estimated class probabilities, Cheng & Cloninger (0222) consider using the average difference in logits (i.e,, activations in the last hidden layer).
We note that the learned classifier in C2ST can be applied to estimations of a density ratio p(a) q(z) also known
as the likelihood ratio trick (Hastic et al. 2001 Sugiyama et al. 2012). Density ratio cstimation has attracted a great deal attention in the statistics and machine learning communities since it can be employed for
estimating divergences between two distributions, such as the Kullback Leibler divergence (Titsias & Ruiz
2019
Huszár 2017 and Pearson divergence (Srivastava al. 2020
2.3
Kernel-based: maximum mean discrepancy (MMD)
MMD is a popular distance metric that is applicable to variety of data domains, including high-dimensiont
continuous data spaces, strings of text, and graphs (Borgwardt et al. 2006 Gretton et al. 2012a Muandet
ee. al. 2017). It has been used to evaluate generative models (Sutherland et al. 2021; Borji, 2019 Lueckmann
et al. 2021 and also has the ability to indicate where the model and the true distribution differ (Lloyd
& Ghahramani 2015). The distance provided by MMD can straightforwardyy be used to test whether the difference between two sets of high-dimennsionll samples scalls significant. (Gretton et al. 2012a).
To assess whether two sets of samples are drawn from the same distribution, MMD makes use of kernel function to (implicitly) embed the samples via embedding function é, also called a feature map. If choose the right kernel, can end up embedding underlying distributions are casily compared. We will motivate the use of the kernel in MMD by illustrating different explicit embeddings before introducing the implicit embedding via a kernel k. Note that this explanation is inspired by SSuherland [2019).
samples in space where the properties of the
In first step, we can define MMD as the difference between the mcans of the embedding of two distributions Pl and p2:
MMD (6. P1. [=)((=)((())]] - (1))]
for any embedding function Φ
If we have samples of real numbers from two distributions pı and p2 (Fig. 6a), there are various embedding functions é we could use to compare these. The simplest possible function σ(), R + R is the identity mapping ()1) (x) = x (Fig 60, left). However, in this case, the MMD will simply be the absolute difference between the means (first moments) of the distributions (for details, see Section A.2
MMD( PI. P1. 1pp Mpal.
This does not yet allow us to discriminate different distributions with equal means (all distributions in Fig. 6 If we now expand our embedding with quadratic term, 2( R - R2 as (x)) = x
x
(Fig 6o, right), the
MMD yields (for details, see Section A.2
MMD? (2) P1] = (Ap. - Mp,)² + (u + + - 1p pz ya.
In this case, we can also distinguish distributions with different variances (or second moments; allowing us differentiate between two out of three distributions in Fig. 6 If we want to distinguish between all could keep adding additional features to to capture higher moments. However,
distinct distributions, this could become infeasible if we want to make sure two probability distributions are exactly equal, ie., have exactly the same moments, we would need to add infinitely many moments. Luckily, there is a trick we can exploit. First, we can rewrite MMD in terms of inner products of features (denoted with (s 1); for details, see Section A.2b as
MMD (6) P1. [aa]==Ep.eee (eo (ppp://(ce))leee))))p.....o.oee.
o(y 2E (o))
We can now rewrite the inner product (o(z), d(x')) in terms of kernel function k: (o(z). k(x, Thus, if can find kernel for our fcature map, avoid explicitly computing the features altogether and instead directly compute
(k, = (eg Epowiss v')l 2Ep, (k))))))))))
8

---

Published in Transactions on Machine Learning Research (08/2024)
a
b
4(x) =
((x)=[x.x]
Pina(X) (x)
+
p, [6]
(x)
E., [+]
P2(X)
*
E [4]
p(x)
p(()
4,00)
X
+(x)
0,00
(xx)
p(x)
(xx) [x. x....
(x)
MMD
kernel trick
Figure 6: Maximum mean discrepancy (MMD). (a) Two example distributions pı(x), (2) and observed data (z) that we want to compare. (b) MMD can be defined as the difference between the expectations of some embedding function ((x). If we take the identity as embedding (z) r; left), we end up computing the differences between the means of the distributions, which are all equal for the three distributions. If we add a quadratic feature (g(2) = [x.x²"' right), we can distinguish distributions with different variances. Note that we still have (M P2. Ptruel O, despite P2 being different from Ptrue (c) Using the kernel trick that capture all relevant features of the distributions.
avoid computing the embeddings all together but use implicit embeddings
Evaluating the kernel function instead of explicitly calculating the features is often called the kernel trick (Fig. (r). If we can define kernel whose corresponding embedding captures all, potentially infinitely many moments, we would have an MMD that is zero only if two distributions are exactly equal and the MMD becomes a metric. These kernels are called charactersstic (Section A.2 Gretton et al. 2012ab), and inchude the commonly used Gaussian kernel: ka(x, x') =xp(-1===(1") A number of other kernels can also be used (see, e.g., Sriperumbudur et al. 2009). For instance, using kernel induced by the Euclidean distance, MMD be shown to be equivalent to the standard energy distance (Székely & Rizzo 2013). In fact, wider equivalence between MMD and the generalized energy distance has been established, using distance-induced kernels (Sejdinovic et al. 2013).
2nt
MMD in practice Typically, the kernel version of MMD is used, which is straightforwardly estimated with its empirical, unbiased, estimate:
MMD
m Σ
Σ kx4,
n n(n - 1)
k(y vy)
2
Σ
k( m5).
mmm
mn
j3
jzi
6.
MMD this form can be applied to many forms of data as long as we can define kernel, which can include
graphs (Vishwanathan et al. 2010
Gärtner 2003) or strings of text (Lodhi al. 2002), in addition
vectors and matrices.
When estimate the MMD with finite number of samples, the selection of the right kernel and its parameters becomes crucial. For example, when using Gaussian kernel, one has to choose the bandwidth o. The MMD approaches zero if we take a to be close to zero (then (x) = 1 if æ = x' else ko(ax - 0) or if a is large (then kz(z, x') + 1 V x, 2'; Gretton et al. 2012a). A common heuristic to remedy this parameter choice is picking the bandwidth based on the scale of the data. The median heuristic sets the bandwidth to the median distance between points the aggregate sample (Gretton et al. 2012a). Another common
approach is based cross- validation, data splitting (Gretton al. 2012a b Jitkrittum al. 2016
9

---

Published in Transactions on Machine Learning Research (08/2024)
dim 2
joint embedding function
D(
dim 2
dim
Figure 7: Network-based metrics. Instead of directly computing distances in data space, complex data, egg,, natural images of dogs sampled from (x) and aircraft sampled from p2()), are jointly embedded into a vector space. The embedding function can, for example, be deep neural network. The resulting distributions in feature space are then compared by classical measure of choice D.
Sutherland et al., 2021 The dataset is divided, with a hold-out set used for kernel selection, and the other part used for evaluating MMD. While the data splitting method docs not involve any hcuristic, it can load to errors in MMD since it reduces the number of data points available for estimating the MMD, Recent work
attempts to choose hyperparameters without employing data splitting (Biggs et al. 2023 Schrab al. 2023 Kübler et al. a))
We often aim for kernel that captures the simiiiityy between the data points well, such kernel can be domain-specific specifically designed for downstream analysis tasks. The similarity between two strings (e.g., DNA sequences or text) can, for instance, be estimated by looking at the frequency of small subsequences (Leslic et al. 2001 Lodhi et al.j 2002). Furthermore, it is possible to aggregate simpler kernels into a more expressive one (Gretton et al.] 2012bb), or to use a deep kernel (i.e., based on neural networks)
that can exploit features of particular data modalities such images (Liu et al. 2020 Gao et al., 2021).
2.4 Network-based: Embedding-space measures
Distribution comparisons on structured data spaces, such as a set of natural images, present unique chal- lenges. Such data is usually high-dimensional (bigl-resolution images) and contains localized correlations. Furthermore, images of difffrrnt object classes (such as airplanes and dogs) share low-level features in the form of edges and textural details but differ in semantic meaning. Similar challenges oceur for time-scries data, natural language text, and other complex data types (Smith & Smith 2020 Jeha et al. 2021
In this section, we rely on the example of natural images, but the presented framework generalizes to other data types. Naive distances would operate on a per-pixel basis, leading to scenarios where, for example, white dogs and black dogs are considered vastly different despite both being categorized as dogs. As we would like to have a distance measure that operates based on details relevant the comparison, we can leverage neural networks trained on large image dataset that captures features ranging from low-level to high-level semantic details: While earlier layers in convolutional neural network focus edge detection, color comparison, and texture detection, later layers learn to detect high-level features, such dog's nose or the wing of an airplanc, which thought to be relevant for a mcaningful comparison. Embedding- based distances use these activations of neural network layers as an embedding to compare the image distributiooss The most popular distance in this class is the Fréchet Inception Distance (FID; Heusel et al. 2017), used to evaluate generative models for images. The FID uses a convolutional neural network's 's embeddings (specifically InceptionV3 (Szegedy et al. 2015)) extract relevant features, applics a Gaussian approximation in the embedding space, and computes the Wasserstein distance this approximation.
10

---

Published in Transactions on Machine Learning Research (08/2024)
A FID-like measure, in essence, requires suitable embedding network f X - R2, where transforms the data from the original high-dimensional space X' into a lower-dimensional, feature-rich representation in Rdd (Fig. 7 Once the data samples are mapped into this reduced space through the embedding network, the two sets of embedded samples can be compared using the appropriate distance. When evaluating generative models for natural images, is common lo approximate the embedded distributions with Gaussian distributions by estimating their respective mean μ and covariances E. Under this Gaussian approximation, the squared Wasserstein distance (also known as the Fréchet distance) can be analytically computed as
(2,,, (42, E2), = Il - -all2 Tr -2
(4)
In principle, any appropriate metric can be used in place of the Fréchet distance. For instance, Jayasumana et al. (2023); Binkowski et al. (2018); Xu et al. (2018) use MMD as metric in the embedding space, and the MMD-based Inception Distance is often referred to as Kernel Inception Distance (KID). KID is known to have some advantages over FID: unlike FID, KTD has a simple, unbiased estimator with better sample efficiency (Jayasumana et al. 2023), and does not assume any parametric forms for the distributions. Since KID involves MMD, we must carefully select the proper kornel and its hyperparameter when applying it. A related and commonly used quality measure for images is the Inception Score (Salimans et al., 2016). In contrast to the FID, this measure uses the average InceptionV3 predicted class probabilities and compares them with the true marginal class distribution. Note that while both this score and the FID can agree with traditional distances (e.g., certain divergences), they might evaluate models differently (Betzalel et al. 2022 see Barratt & Sharma 2018)) for further limitations of the Inception Score.
One of the biggest limitations is the requirement of a suitable embedding network. Newer and more robust networks, such as the image network of the CLIP (Radford et al., 2021 vision-language model, provide better and more semantically consistent embeddings (Belzalel al. 2022 Jayasumana el al., 2023) than the InceptionV33 network. However, as the embedding network is generally non- -oj-injecttvee identical distributions in the embedding space may not necessarily translate to identical distributions in the original space. Previous research has demonstrated the FID's sensitivity to preprocessing such as image resizing and compression (Parmar et al., 2022). Additionally, FID estimates are biased for finite sample sizes, making comparisons unreliable due to dependency on the generative model. However, methods to obtain a more unbiased estimate have been proposed (FID bo: Chong & Forsyth 2020 Betzalel et al. 2022
Limitations
3 Comparing distances: sample size, dimensionality, and more
In this section, we empirically compare these distances along several dimensions, with additional results presented in the Appendices. When evaluating (or training) generative models, is important to understand that different statistical distances pay attention to different features of the generated samples. A key aspect to consider is that differences between distances can be especially pronounced in applications where we only have a limited amount of data points, e.g., identifying rare cell types (Marouf et al. 2020). where we have very high-dimensional data, e.g., neural population recordings in neuroscience (Stringer et al. 2019). In such cases, one needs to ensure that the distance measures can reliably distinguish different distributions for the given sample set size while remaining computationally tractable. We therefore perform a number of empirical studies here: First, we investigate SW, C2ST and MMD when it comes to distinguishing data sets given varying numbers of samples and data with varying dimensionality (Section 3.1 Second, we investigate the scaling properties of FID on the ImageNet dataset (Section 3.2), as well considering the other distances when computed the InceptionV3 image embeddings. We also summarize the theoretical sample and computational complexities of the four discussed distances in Section A.6.
Furthermore, different distances might weigh differently the importance of sample diversity, or mode coverage, versus how well the modes of the true distribution are captured. If models only capture one mode, or a subset of the modes, i.c,, exhibit mode collapse, they might have learned to generate realistic but unvaried samples.
We illustrated the trade-off between mode collapse and mode coverage by optimizing mis-specified (Fig. and well-specified (Fig. 53) models using the different distances (see also Section A.7 and Theis et al. 2016
11

---

Published in Transactions on Machine Learning Research (08/2024)
a
2-dim Mixture of Gaussians
10-dim Gaussian
n-dim Gaussian
true approx.
0.4
0.4
true shifted
0.50
SW
SW
0.25
0.2
0.2
shifted
0.00
0.0
0.0
1.0
1.0
1.0
C2ST
approx
C2ST
shifted
C2ST
shifted
0.5
0.5
0.5
0.01
0.05
true approx.
true shifted
MMD,
MMD
0.02
MMD
true
shifted
0.00
0.00
2000
4000
2000
4000
10'
10"
10"
sample size
sample size
dimensions
Figure 8: Scalability of different statistical distances with sample size and dimensionality. (a,b) Comparison of sample sets with varying sample size (between 50 and 4k samples per set) of a 'true' distribution, either with a second set of samples of the same distribution or with a sample set from a model consisting of an approximated/shinn /shiftte distribution. We show the mean and standard deviation over five runs of randomly sampled data. Note that the subscript for MMD distances (bottom) denotes the bandwidth of the Gaussian kernel used for a given dataset and we report the squared distance for MMD. (a) Distances for the 2d-MoG example shown in Fig. compared to samples from uni-modal Gaussian approximation with the same mean and covariance. (b) Distances for a ten-dimensional standard normal distribution, with model whose first dimension is shifted by one. (c) Distances based on 10k samples from slandard normal distribution with varying dimensions (between 5 and 1000). As in (b), the first dimension of the model is shifted by one. We show the mcan and standard deviation over five runs of randomly sampled data. One MMD bandwidth was selected for all n-dimensional datasets.
Note that the absolute values of the distance measures can be hard to interpret and different measures lie on different scales. For this reason, compare the distance between set of sample from the model and set. of samples from the true distribution to the distance between two sets of samples from the true distribution.
3.1 Varying number of samples and dimensionality
Here, we assessed SW, C2ST and MMD using three datasots: First, we compared the two-dimensional Mixture of Gaussians (*2d-MoG") dataset introduced in Fig. with samples from a model consisting of unimodal Gaussian approximation with the same mean and covariance the true data. Second, we compared samples from a ten-dimmen cen-dimensional standard normal distribution with model that is also normal distribution that matches the true distribution in all but the first dimension-wheet the mean is shifted by one (*10-dim Gaussian"). Last, we repeated the second experiment, now taking a normal distribution with varying dimensionality as true dataset, and a model who again matches in all dimensions, except for the shifted first dimension (*n-dim Gaussian"). We used default parameters for the SW and C2ST measures while adjusting the bandwidth parameter for the MMD measure for each of the three comparisons.
Sample size We explored the robustness of the distances to low sample sizes on the 2d-MoG and the 10-dim Gaussian dataset. We found that for the 24-MoG dataset, computed values quickly stabilized for
12

---

Published in Transactions on Machine Learning Research (08/2024)
all measures by sample size of 1000 and yielded the expected results of indicating low differences between two sets of samples from the true distribution (frue), and a higher difference wetween samples from the true and model distributions (approx.). All measures failed to reflect the dissimilarity of the distributions at the lowest sample size of 50 samples (Fig. 8p). Here C2ST's behavior differs from MMD and SW, with C2ST indicating that the dist.ibuttons are similar (C2ST Ri 0.5) while the other two distances indicate they are different (distance #0).
For the 10-dim Gaussian dataset, we observed that all three distances can robustly identify samples from the same distribution as more similar than samples from different distributions with no measure being clearly superior to the others (Fig. pp). For the previous 2d-MoG experiment, more samples were required to clearly detect the difference between the two distributions (Fig. 8a) as compared to the 10-dim Gaussian (Fig. (p).
Intuitively, the more pronounced the differences in the distributions we compare, the fewer samples we need to detect these differences (see additional experiments in Supp. Fig. 54
Dimensionality We further tested how the distances scale with the data dimension using the n-dim Gaussian dataset. As the dimensionality increases, all distances indicate no difference between two sets of samples from the true distribution, except C2ST which is the only mcasure that consistently identifies samples from the true and model as distinct (Fig. D When we changed the structure of the data distribution (e-g,, by changing the mean of all dimensions or their variances, see Supp. Fig. S5), we observed a similar picture with some particularities: While the SW distance with a fixed number of slices has difficulties if the disparity between the distribution only in one dimension, its performance drastically improves for differences in all dimensions, which is expected from the random projections SW is performing. C2ST seems to robustly detect differences even in high dimensions in these modified datasets, though previous experiments showed that this measure can be oversensitive to small changes in high dimensions (Fig. p.c). Lastly, while here MMD is not robust across different dimensions for a Gaussian kernel with fired bandwidth, one typically chooses the bandwidth to be on the order of the distance between datapoints (e.g., using the median heuristic; Section 2.3). Note that in general, MMD can be highly sensitive to kernel- and hyperparameter choice; an appropriate setting depends not only on the dimensionality of the data (Supp. Fig. S6 S7 and (8). but also on the structure of the distributions (Supp. Fig. (55). SW distance, on the other hand, appeared robust to number of random projections used (Supp. Fig. S1 and A.4
3.2 FID-like distance comparison on ImageNet
To explore scaling properties of FID-like distances, we generated high-quality synthetic samples using state-of-the-arl diffusion model as described by Dockhorn et al. (002)), which we compared to images from the ImageNet dataset, which contains 1000 classes, 100 images per class, All images were embedded using
the pre-trained InceptionV33 network (Deng et al. 2009 Szegedy et al.] 2016). following the implementation of Hensel et al. (017)) transforming the raw images into a 2048-dimensional feature space.
Sample size We first produced varying number of samples with the base unconditional version of the diffusion model and compared those to samples from Imagenet (Fig. 9h). Calculating the FID involves computing the mean and covariance of the distributions in the embedding space and then calculating the squared Wasserstein distance analytically. However, we broadened our evaluation by applying additional distances to the distributions in the embedding space. While SW distance, C2ST, and FID effectively highlight the greater dissimilarity of synthetic samples to real images (i.e., the ImageNet test set) even for low sample sizes, the distinctiveness of the FTD only becomes apparent when analyzing more than 2000 samples. This is in line with common implementations which generally recommend using least 2048 samples (Heusel et al. 2018a b). Another reason for the limited performance could be that the Gaussian assumption in the FID can be violated (Jayasumana et al. 2023 Betzalel et al. 2022). In contrast, the other distances reliably estimate a larger inter-datasct distance in regimes with few samples.
Class discriminability We aimed to determine the effectiveness of various distances in discriminating between images from different classes. To this end, we focused on comparing ImageNet images of dogs (D) with ImageNet images of other (non-dog) classes (-D; Huang et al. 2021). All investigated distances, except FID, were successful in identifying images from different. classes as being more distinct than images from the
13

---

Published in Transactions on Machine Learning Research (08/2024)
a
1,0
0.03
1.0
0.04
true generated
accona) all classes
generated
SW
50-
0.75
0.02
0.01
Subcet000500) clanses
0.01
0.00
0.5
10"
0.0002
0.000
susse)]lasses)
subseniciassso) all plasses
genersted
getaadded
dassess
MM0
0.0001
12 16
50
10"
0.0000
0-
5000
10000
5000
10000
Da
-0
Dv
Do
-0
1000
500
1000
sample
-0 -0
-0
D
-D
classes
Figure 9: Comparison of distances for ImageNet. (a) A comparison between the ImageNet test set and samples generated by an unconditional diffussin model with varying sample sizes. (b) Distance evaluation on dog classes (D) versus non-dog classes (-D), highlighting differences in image representation between these two categories of real data. (c) Distances between sets of randomly selected images vs. varying number of included classes of images (from 10 to 1,000) from the test set, using synthotic samples created by a conditional diffusion model.
same class. The FID comparison between data with multiple classes (-D vs. -D) is higher than across dog classes and other classes (D vs. -D). This indicates that FID may be comparably insulficient in recognizing dissimilarities between distributions that have clear distinctions (whether across individual classes or in a one vs. all scenario).
Mode coverage To examine the effects when only considering a subset of classes from the dataset (i.e., modes of the distribution), we used the diffusion model to generate class-conditional images matching the classses of the ImageNet test set (Fig. 9c). Our analysis involved comparing the complete test set against model-generated datasets that included only subsets of classes. For comparison, we created control dataset by randomly sampling from all classes (dashed curve in Fig. D+). This approach revealed that limiting the dataset to a small number of classes compromised the performance across all evaluated distances, in contrast to the outcomes observed when randomly excluding a subset of images -eee-iniiiiiingggeeeeee that all distances successfully detected the lack of mode coverage. To achieve performance comparable to that observed with random removals, it was necessary to include at least 800 classes in the comparison. As the InceptionV3 network is, in essence, trained to classify ImageNet images (Szegedy et al. 20155; under certain regularization schemes), the extracted high-level features may also be very sensitive to class-dependent image features and not necessarily for general image quality. This behavior can be observed in Fig. gh and was recently explored by Kynkäänniemi et al. (2023). By replacing InceptionV3 with other embedding networks (e.g., CLIP, which is trained to match images to corresponding text), this class scnsitivity could be reducod (Kynkäānniemi et al. 2023
Additional generative models We additionally generated images using consistency model (CM) for unconditional image generation (Song et al.] 2023), to investigate how these metrics perform when evaluated on images created by different generative models. This model was trained on ImageNet 64x64, similar to the GENIE model (Dockhorn et al. 2022). Morcover, we included the following models: BigGAN (Brock et al. 2019), ablated diffusion model (ADM; Dhariwal & Nichol 2021), Glide (Nichol et al. 2022), Vector Quantized Diffusion Model (VQDM; Gu et al. 2022), Wukong @Wukong 2022), Stable diffusion 1.5 (SD1.5; Rombach
ct al. 2022b) and Midjourncy (Midjourney 2022; details in Appendix A.13). We evaluated the metrics (and multiple commonly used variants of KID and C2ST) for each model against the ImageNet test set (see
Table S4
While there is some agreement between the metrics regarding which model generates images closest the ImageNet test set, there are also differences in the relative ordering across different metries. As expected, the most recent. unconditional models trained directly on ImagenNet 64x64 performed best in our
14

---

Published in Transactions on Machine Learning Research (08/2024)
b
decision-boondaries
caidence
C2ST
0.50
MID
0.48
0.45
non-deciaion
time
deciaion
DOM1 COM2
heel
leel
leel
DDM1
leal
heal
DDM1
Inain
leal
heal
leal
DDM1
12g
10g
Use
vs test
vs
106t
1est
DDM1
DDM2
test
1est
DOM1
DDM2
DDM2
teet
DOM1
DOM2
DDM2
Figure 10: Comparing models of primate decision making. (a) Schematic of Drift-Diffusion model (DDM), a classical neuroscientific model of decision making behavior. Overall, evidence drives the model toward one of two choices (drift), but sensory and environmental noise result in random fluctuations in evidence integration (diffusion). (b) Distributions of primate decision times from the test set (black), and two fitted models of varying complexity: DDM1 (gray) and DDM2 (gold). (c) SW distance, C2ST, MMD (bandwidth=0..) between subsets of the generated and real data distributions. FID not applicable these comparisons, because the data are one-dimensional distributions. peatter-ponnte indicate comparisons between ten random subsets from each datasct. Thick horizontal bars indicate median values.
evaluation (GENIE, CS), better than the two other unconditional generative models (BigGAN, ADM). The other models are text-to-image and thus only prompted to generate images from specific ImageNet classes (GLIDE, VQDM, Wukong, SD1.5, Midjourney). Interestingly, the prompted models performed better than older unconditional models (BigGAN, ADM) most of the time. Recall that all we evaluate is the similarity the ImageNet test sct; prompted versions might produce images from the correct classcs but might contain differences in style or appearance compared to actual images in ImageNet. Despite the demonstrated thus also sensitive to different "styles" of natural images. Note that in this case, being closer to ImageNet does not necessarily mean generating better images (based on human perception), but rather creating images that are more ImageNet-like.
class-sensitivity (Fig. g 0p), the InceptionV3 embeddings
4 Scientific applications
To demonstrate how the presented distances apply to evaluating generative models of scientific applications, focus here two examples: decision modeling in cognitive neuroscience and medical imaging. For cach application, we used two generalive models or simulators to sample synthetic dała. We then compared the synthetic samples to real data (hold-out test set) using the discussed distances. To obtain baseline values for each distance, we computed distances between subsets of real data. For SW distancc, MMD, and FID we anticipated values proximal to zero, while for the C2ST, we expected a value around 0.5. These baseline assessments provide a lower threshold of model fidelity which we compared the deviation of model-generated samples.
4.1 Models of primate decision making
We explored the fidelity of two generative models in replicating primate decision times during a motion- discrimination task (Roitman & Shadlen 2002). We evaluated two versions of a Drift-Diffusson Model (DDM; Fig. 10p; Ratcliff 1978), frequently used model in cognitive neuroscience. The two versions differ with respect to the drift rate, which is the speed and direction at which evidence accumulates towards a decision, and the decision boundaries, which determine how much evidence is needed to make decision. Specifically, the first version (DDM1) uses drift rate that varies linearly with position and time and decision boundaries that decay exponentially over time, whereas the sccond version (DDM2) uses drift rate and decision boundaries that are constant over time (for details, see Section A.14). We fitled each model to empirical primate decision times with the use of the PyDDM toolbox (Shinn et al. 2020), generated one-dimensional synthetic datascts, and compared each dataset to the actual primate decision time distributions. While the resulting distributions of decision times are visually similar (Fig. 10p), the DDM- -oM-egeertted distributions DDM1 and DDM2 are noticeably broader compared to the more tightly clustered real decision times. Moreover, the
15

---

Published in Transactions on Machine Learning Research (08/2024)
a
b
512D
PGGAN
Stable Diffusion
C
x10"
x10
1,00
1.5
MMD
1.0
SW
0.75
0.5
0.50
train
test
tast
tast PGGAN
train
test
1est
1est PGGAN
train
lest
lest
test PGGAN
train
1est
1est test PGGAN
PGGAN
Figure 11: Comparing generated and real X-ray images. (a) Sketch of the embedded distributions of X-ray images from the three different datasets: test set of real dataset (black), Progressive Growing Generative Adversarial Network (PGGAN) (grey), and Stable Diffusion (SD) model (gold). (b) Examples from real and generated X-ray images. Three full-view examples from each distribution and four examples magnifying the top right corner. (c) SW distance, C2ST, MMD (bandwidth=50), and FID between samples of the real and generated distributions of embedded X-ray images. ceatter-points indicate comparisons between ten random subsets from each dataset. Thick horizontal bars indicate median values.
DDMI distribution appears more similar to the real distribution than that of DDM2, which is shifted towards the left. As expected, the DDM1 model more precisely mimics the real data distribution, as compared to the DDM2, across the median values of the SW distance, MMD, and C2ST distances (Fig. [0p).. For C2ST, DDMI and the real data distribution are even indistinguishable, with median C2ST values around 0.5. This suggests that SW and MMD provide more nuanced differentiation between the models.
4.2 Chest X-ray image generation
In the second application we turned to high-dimensional example, in which we compared synthetic X-ray images generated by Progressive Growing Generative Adversarial Network (PGGAN) model (Segal al., 2021 and by StableDiffusion (SD) model (Malik & Humair 2023) to real chest X-ray images from the ChestX-ray14 dataset (Wang et al. 2017). Each image has Lotal dimension of 1024 x 1024 pixels.
From visual inspection, we note two observations: First, the images produced by the SD model are clearer and sharper than either the real images or those generated by the PGGAN. Second, generated images contain unrealistic artifacts that distinguish them from real X-ray images (Fig. T1p). In real images, the top often contains annotations including, e.g., patient id, side of the body, or the date the X-ray was taken. These textual elements often contain artifacts or, in case of SD, are completely unrealistic. To compare these high-dimensional images, we embedded them in 512-dimensional embedding space using the CheXzero network TTu et al. 2022), a CLIP (Radford et al. 2021 network fine-tuned for chest X-ray images. We opted for using this specialized network instead of the standard InceptionV3 network as might overcome biases introduced by classification task training |Kynkäänniemi et al. 2023)
As expected, samples generated by PGGAN are closer the real data across all distances compared SD-generated data (Fig. [Tp), likely due unrealistic sharpness and more obvious textual artifacts of the SD-generated images. However, C2ST is even high between PGGAN outputs and the real data, suggesting
16

---

Published in Transactions on Machine Learning Research (08/2024)
that the tigh-diimessionalityy of the data increases the sensitivity of this measure. Taken together, our results suggest that PGGAN is more accurate in generating realistic X-ray images compared to SD
Our findings show that using different metries can support different conclusions, For instance, C2ST suggests equality between DDM1 and real decision time data, whereas SW distance and MMD metrics indicate a larger difference between DDM1 and the real data. Similarly, in analyzing X-ray image generation, SW distance, MMD, and FID metrics suggest high similarity between indicates strong difference. Thus, we want to highlight the importance of using multiple complementary distances for best results and understanding of model limitations.
and real images, whereas C2ST
5 Discussion
This work describes and characterizes four commonly applied sample-based distances representing different methodologies for defining statistical distance: Using low-dimensional projections (SW), obtaining a distance using classifiers (C2ST); using embeddings through kernels (MMD) or neural networks (FID). Despite their operational differences, they are all based on a fundamental concept: simplifying compler distributions into more manageable feature representations to facilitate comparison. Sliced distances effectively reduce multidimensional distributions to a set of one-dimensional distributions, where classical metrics are more casily applied calculated. MMD uses kernels to (implicitly) project samples into higher dimensional feature space, in which comparing mean values becomes more expressive. Classifier-based methods (C2ST) transform the task of distribution comparison into a classification problem; comparison is made by investigating how well a classifier can distinguish the distributions. Lastly, network- based distances, such as FID, explicitly map samples into a representative feature space and compare distributions directly within this space.
In the paragraphs below, we highlight the features and limitations of these investigated distances. Additionally, we discuss the relationships between these metrics and connect them to current related work.
Sliced Distances Sliced distances stand out for their computational efficiency in evaluating distributional discrepancies. However, when distributions differ primarily in lower-dimensional subspaces, sliced distances might not detect these subtle differences without a large number of slices (see Fig. [p.. There are approaches to reduce this effect by considering other projections than simple linear slices, as described in Section 2.1 In our experiments, the metric did show convincing results and contrast to the MMD, C2ST, and FID, SW distance does not require one to choose specific hyperparameters (for which results can differ drastically). Although currently not extensively used in literature for evaluation, this makes the SW distance efficient, scalable, and consistent baseline for general distribution comparisons. Yel, this also makes it less flexible to adapt to specific features of interest. The Wasserstein and SW distances are not interpretable, and admit only biased sample-basod cstimates. This can be a limitation for some tasks. However, due to its computational efficiency and differentiability, the SW distance is commonly used as a loss function to train generative models, such as GANs (Deshpande et al., [0118] Wu et al.) 2019 Autoencoders (Wu et al. 2019), nonparametric flows Liutkus et al.) 2019), normalizing flows (Dai & Seljak 2021), and multi-layer perceptrons (Vetter et al.
2024
Although the majority of research on sliced distances focuses on sliced Wasserstein metrics, slicing oth metrics is also possible. For a certain subset of choices, equivalence to MMDs can be established (Feydy
et al.
2019 Kolouri et al. 2019 Hertrich et al.] 2024
Classifier Two-Sample Test (C2ST) C2ST distinguishes itself by producing an interpretable value: classification accuracy. This characteristic makes C2ST particularly appealing for practical applications, as it is easy to explain and interpret. A notable drawback is the computational demand associated with training classifier, which can be substantial. Moreover, C2ST's effectiveness is critically dependent on the selection and training of a suitable classifier. Interpreting results reported for C2ST requires knowledge of the classifier used and its appropriateness for the data at hand. Furthermore, automated training pipclines may encounter failures, such as when the trained classifier performs worse than chance, often due to overfitting lo cross-validation folds (see also Section A..)). On the other hand, it is able to even detect subtle differences within two distributions in high dimensions. Even if there is a difference in only a single out of a thousand dimensions (for which SW distance and MMD might struggle), C2ST is able detect it (see Fig. Bc). This might be desirable, but can also be problematic. When comparing images, slight variations in few pixels
17

---

Published in Transactions on Machine Learning Research (08/2024)
may not be visually noticeable, potentially making them unimportant to the researcher. In high-dimensional complex data, such slight variations are quite likely. Thus C2ST can be close to 1.0 in the high-dimensional setting, making it practically useless for evaluation (see Fig. 5p. [1p]. The C2ST can be shown to be a MMD with a specific kernel function parameterized by the classifier Liu et al. 2020
MMD The Maximum Mcan Discrepancy is strong tool for comparing two groups of data by looking at their average values in a special feature space. The effectiveness of MMD largely depends on the kernel function chosen (implicitly representing the feature space), which affects how well it can spot differences between various types of data. Inappropriate kernel choice can leave the metric insensitive to subtle differences in the distribution |Gretton et al. 2012b Sriperumbudur et al. 2009 see Fig. S The MMD can be estimated efficiently and is differentiable, and thus often used as a loss function for training generative models (Dziugaite et al. 2015 Arbel et al. 2019: Li et al. 2017: Binkowski et al.) 2018 Briol et al., 2019). Yet, a kernel must satisty certain criteria, c.g., positive definiteness, making the design of new kernel functions challenging. Such constraints are relaxed for FID-like metrics, which focus on explicil representations of the embedding, whereas (kernel) MMD instead focuses on implicit representations. One advantage, however, is that the implicit embedding allows for infinite dimensional feature spaces (through characteristic kernel functions). These can be proven to be able to discriminate any two distinct distributions, something that is impossible through explicit representations used by the FID. Recently, Kübler et al. (2022a) proposed a method to estimate MMD via a witness function that determines MMD (Appendix A.2 This method is closely related to C2ST
in that both estimate a discrepancy among distribut ions via classifier (Kübler et al. 2022a Section 5).
Network-based Network- approaches for evaluating distributions focus on the analysis of complex data, emphasizing the importance of capturing high-level, semantically meaningful features. These methods leverage neural networks to project data into a lower-dimensional, feature-rich space where traditional statistical distances can be applied more effectively. This is particularly important for tasks where the visual or semantic quality of the data is important, making them a popular choice for assessing generative models in domains such as image and text generation. The primary challenge lies the design of suitable network architectures that can extract relevant features for accurate distribution comparison. Even more important than for the C2ST, this network must. be well-established and shared which is a necessary but not sufficient. criterion (Chong & Forsyth 2020) to compare different results. While such well-established defaults exist for images (Szegedy et al.] 2015; Radford et al. 2021), this is not the case for other domains. For example, the time series generation community did not yet establish a default, and embedding networks are either trained or chosen by the authors {Smith & Smith 2020 Jeha et al. 2021). We demonstrated that the sensiivvy of FID (Section 3.2) often leads to model collapse, as seen in GANs. However, it may not accurately reflect the overall image quality. For example, Betzalel et al.] (2022); Kynkäänniemi et al. (2023); Jayasumana (2023 found that relevant features sometimes can disagree with human judgment and that CLIP et al. embeddings align more closely to what humans perceive as favorable unfavorable. Yet, FID features have been shown to align much better with human perception than traditional metrics (Zhang et al. 2018 Recent developments in networks based approaches include the use of Central Kernel Alignment (CKA; Cortes et al. 2012) to compute the distance between network-embeddded samples. CKA scores show considerable stability when evaluated with different choices of network architect ures and layers (Yang el al. 2023). Another newly introduced metric, Mauve (Pillutla et al., 2021 Liu et al.) [002 Pillutia et al.] 2023). , can, for instance, be used measure how close geneerte-eeeeeteed text to human language using an external language model to embed the samples from each distribution. This metric uses divergence frontiers to take into account the trade-off between quality and diversity when evaluating generative models.
Closing remarks Ultimately, the choice of distance hinges the nature of the data under consideration and the specific characteristics one aims to compare. Given a particular dataset and problem, it may be necessary to look beyond the distances discussed in this paper. For instance, in the realm of human-centric data such as images and audio, pal indistinguishability of distributions is crucial (Gerhard et al.) 2013) Zhang et al. 2018). Time series data, with its inherent temporal structure, requires metrics that accommodate temporal shifts and variations without disproportionately penalizing minor discrepancies in timing, such as Dynamic Time Warping (uüller (007) or frequency- methods (Hess et al. 2023). Additional recent works propose new distances based on fields such topology (Barannikov et al. 2021
18

---

Published in Transactions on Machine Learning Research (08/2024)
In general, regardless of the specific use case, it is advisable to use multiple distance mcasures to obtain comprehensive view, as relying on a single measure may lead to competing conclusions about the model under evaluation.
Throughout this paper, we have explained and analyzed four approaches to measuring statistical distance. While this represents only a small subset of all possible measures, we hope have provided the foundational knowledge researchers need to find, understand, and interpret statistical distances relevant to their own scientific applications.
Code availability
All code for replicating and running our analysis is available at: hntpp://github..e http./////macaaaaaaaab.aaaaae Note that for implementations of the C2ST we largely made use of those from the simulation bwsed- package (Tejero-Canne Cantero et al. 2020
Author Contributions
Conceptualization: AD, AS, CS, GM, JKL, LH, MD, MG, MP, SB, ZS
Software: AD, AES, AS, FG, FP, GM, JK, JKT, JV, LH, MD, MG, MP, RG, SB, 78 Draft: AS, CS, FG, FP, GM, JK, JV, LH, MD, MG, MP, RG, RR, SB, ZS Writing Review & Editing: AD, AS, CS, FP, GM, JHM, JKL, JV, LH, LU, MG, MP, RG, SB, ST, ZS Visualization: AD, AES, AS, FP, GM, JKL, JV, MD, MG, MP, RG, RR, ZS Project administration: AS, MG, MP, SB Funding acquisition: JHM
Acknowledgments
We thank Eiki Shimizu and Nastya Krouglova for reviewing an carly version of this paper. This project was inspired by a similarly structured project by the research group led by Karim Jerbi.
This work was supported by the German Research Foundation (DFG) through Germany's Excellence Strategy (EXC-Number 2064/1, PN 390727645) and SFB1233 (PN 276693517), SFB 1089 (PN 227953431), SPP2041 (PN 34721065), the German Federal Ministry of Education and Research (Tübingen AI Center, FKZ: 011S18039; DeepHHumanVision FKZ: 031T.197B; Simalesam: FKZ 01 (S/20055A--B) the Carl Zeiss Foundation (Certification and Foundations of Safe Machine Learning Systems in Healthcare), the Else Kröner Fresenius Stiftung (Project ClinbrAIn), and the European Union (ERC, DeepCoMechTomer 101089288). ST was partially supported by Grant-in-Aid for Research Activity Start-up 23K19966, Grant-in-Aid for Farly-Career Scientists 24K20750 and JST CREST JPMJCR2015. SB, MD, MG, JK, JL, GM, MP, RR, AS, ZS and JV are members of the International Max Planck Research School for Intelligent Systems (IMPRS-IS).
References
M. Arbel, A. Korba, A. Salim, and A. Gretton. Maximum mean discrepancy gradient flow. Conference Neural Information Processing Systems, 2019.
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, 2017.
S. Barannikov, I. Trofimov, G. Sotnikov, E. Trimbach, A. Korotin, A. Filippov, and E. Burnaev. Manifold topology divergence: framework for comparing data manifolds. Advances in neural information processing systems, 2021.
S. Barratt and R. Sharma. A note on the inception score. orXiv preprint
00100199111 2018
M. Bashiri, E. Walker, K.-K. Lurz, A. Jagadish, T. Muhammad, Z. Ding, Z. Ding, A. Tolias, and F. Sinz. A flow-based latent state generative model of neural population responses to natural images. In Advances in Neural Information Processing Systems, 2021.
19

---

Published in Transactions on Machine Learning Research (08/2024)
M. Basseville. Divergence measures for statistical data processing annotated bibliography. Signal Processing, 2013.
E. Bernton, P. E. Jacob, M. Gerber, and C. P. Robert. On parameter estimation with the Wasserstein distance. Information and Inference: journal of the IMA, 2019.
E. Betzalel, C. Penso, A. Navon, and E. Fetaya. A study on the evaluation of generative models. arXiv preprint 22x: 10935, 2022.
A. Bharti, M. Nasliduyk, O. Key, S. Kaski, and F.-X. Briol. Optinnlly-weighteed estimators of the maximum mean discrepancy for likelihood-free inforence. In Proceedings of the 40th International Conference on Machine Learning, 2023.
F. Biggs, A. Schrab, and A. Gretton. MMD-Fusc: Learning and combining kernels for two-sample testing without data splitting, In Thirly-Seveulh Conference on Newral Information Processing Systems, 2023.
C. M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
M. Binkoowki, D. J. Sutherland, M. Arbel, and A. Gretton. Demystifying MMD GANs. International Conference on Learning Representations, 2018.
D. A. Bodenham and Y. Kawahara. euMMD: Efficiently computing the MMD two-sample test statistic for univariate data. Statistics and Computing, 2023.
S. Bond-Taylor, A. Leach, Y. Long, and C. G. Willcocks. Deep generative modelling: A comparative review of VAEs, GANs, normalizing flows, energy-based and antoregressive models. IEEE transactions on pattern analysis and machine intelligence, 2021.
N. Bonneel, J. Rabin, G. Peyré, and H. Pfister. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 2015.
K. Borgwardt, A. Gretton, M. Rasch, H. Kriegel, B. Schölkopf, and A. Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 2006.
A. Borji. Pros and cons of GAN evaluation measures. Computer Vision and Image Understanding, 2019.
G. F. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
M. Brenner, F. Hess, J. M. Mikhacil, L. Bereska, Z. Monfared, P.-. Kuo, and D. Durstewitz. Tractable dendritic RNNs for reconstructing nonlinear dynamical systems. International Conference on Machine Learning, 2022.
F.-X. Briol, A. Barp, A. B. Duncan, and M. Girolami. Statistical inference for generative models with maximum mean discrepancy. arXiv preprint ar xix: 05944, 2019.
Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2019.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are fow-shot learners. Advances in Neural Information Processing Systems, 2020.
E. Cazelles, A. Robert, and F. Tobar. The W'sssastciinFouurae distance for stationary time scries. IEEE Transactions on Signal Processing, 2021.
T. Che, Y. Li, A. P. Jacob, Y. Bengio, and W. Li. Mode regularized generative adversarial networks International Conference on Learning Representations, 2017.
X. Cheng and A. Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEFF Transactions on Information Theory, 2022.
20

---

Published in Transactions on Machine Learning Research (08/2024)
X. Cheng and Y. Xic. Neural tangent kernel maximum mean discrepancy, In Advances in Neurol Information Processing Systems, 2021.
M. J. Chong and D. Forsyth. Effectively unbiased FID and inception score and where find them. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
C. Cortes, M. Mohri, and A. Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 2012.
B. Dai and U. Seljak. Sliced iterative normalizing flows. In Proceedings of the 38th International Conference on Machine Learning, 2021.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image databasc. In 2009 IEEE conference on computer vision and pattern recognation, 2009.
Deshpande, Z. Zhang, and A. G. Schwing. Generative modeling using the Sliced- W Wasserstein distance. In Proceedings of the IEEE conference computer vision and pattern recognition, 2018.
Deshpande, Y.-T. Hu, R. Sun, A. Pyrros, Siddiqui, S. Koyejo, Z. Zhao, D. Forsyth, and A. G. Schwing. Max-sliced Wasserstein distance and its use for GANs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, 2019.
P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 2021.
T. Dockhorn, A. Vahdat, and K. Kreis. GENIE: Higher-order denoising diffusion solvers. In Advonces in Neural Information Processing Systems, 2022.
C. Du, T. Li, T. Pang, S. Yan, and M. Lin. Nonparametric generative modeling with conditional Sliced- Wasserstein flows. International Conference on Machine Learning, 2023.
G. K. Dziugaite, D. M. Roy, and Z. Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. Proceedings of the Thirty-First Conference Uncertointy in Artificial Intelligence, 2015.
J. Feydy, T. Séjourné, F.-X. Vialard, S.-i. Amari, A. Trouvé, and G. Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, 2019.
Fournier and A. Guillin, On the rate of convergence in wasserstein distance of the empirical measure. Probability Theory and Related Fields, 2015.
J. H. Friedman. On multivariate goodness of fit and two sample testing. eConf, 2003.
K. Fukumizu, A. Gretton, B. Schölkopf, and B. K. Sriperumbudur. Characteristic kernels on groups and semigroups. In Advances in Neural Information Processing Systems, 2008.
Gao, Liu, J. Zhang, B. Han, T. Liu, G. Niu, and M. Sugiyama. Maximum mean discrepancy test is aware of adversarial attacks. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Gärtner. A survey of kernels for structured data. ACM SIGKDD explorations newsletter, 2003.
H. E. Gerhard, F. A. Wichmann, and M. Bethge. How sensitive is the human visual system to the local statistics of natural images? PLOS Compulational Biology, 2013.
P. Ghosal and B. Sen. Multivariate ranks and quantiles using optimal transport: Consistency, rates and nonparametric testing. The Annals of Slatislics, 2019.
A. Gibbs and F. E. Su. On choosing and bounding probability metrics. International Statistical Review Revue Internationale de Statistique, 2002.
21

---

Published in Transactions on Machine Learning Research (08/2024)
Z. Goldfeld and K. Greenewald. Sliced mutual information: A scalable measure of statistical dependence. Aduances in Newral Information Processing Systems, 2021.
Goodfellow, J. Hougel-Abadinn M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances Neural Information Processing Systems, 2014.
A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 2012a.
A. Gretton, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. Pontil, K. Fukumizu, and B. K. Sriperumbudur. Optimal kernel choice for large-scale (wo-sample lests. Advances in Neural Informalion Processing Systems, 2012b.
S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo. Vector quantized diffusion model for tomag- synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
J. Gui, Z. Sun, Y. Wen, D. Tao, and J. Ye. A review on generative adversarial networks: Algorithms, theory, and applications. IEEE Transactions on Knowledge and Data Engineering, 2023.
T. Hastie, R. Tibshirani, and Friedman. The Elements of Statislical Learning. Springer Series in Statistics. Springer New York Inc., 2001.
S. Helgason. The Radon Transform. Progress in Mathematics Birkhääser. Birkhäuser, 1980. ISBN 9783764330064.
J. Hertrich, C. Wald, F. Altekrüger, and P. Hagemann. Generative sliced MMD flows with riesz kernels, In The Twelfth International Conference on Learning Representations, 2024.
F. Hess, Z. Monfared, M. Brenner, and D. Durstewitz. Generalized teacher forcing for learning chaotic dynamics. International Conference on Machine Learnin, 2023.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter. GANs trained by two time-scale update rule converge to nash equilibrium. CoRR, 2017.
M. Heusel, H. Ramsauer, T. Unterthiner, and B. Nessler. FID score for PyTorch, 2018a.
M. Heusel, H. Ramsauer, T. Unterthiner, and B. Nessler. Two time-scale update rule for training GANs, 2018b.
G. E. Hinton, S. Osindero, and Y.W. Tch. A fast learning algorithm for deep belief nets. Newral computation, 2006.
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
A. L. Hodgkin and A. F. Huxley. Currents carried by sodium and potassium ions through the membrane of the giant axon of Loligo. The journal of Physiooogy, 1952.
H. Huang, Z. Li, L. Wang, S. Chen, X. Zhou, and B. Dong. Feature space singularity for out-of-distributiion detection. In SafeAT@AAAT, 2021.
F. Huszár. Variational inference using implicit distributions. Xix preprint arXiv: 1rXiu:1702.0822 082308235.... 2017.
M. Jacobs, B. W. Brunton, S. L. Brunton, J. N. Kutz, and R. V. Raut. HyperSINDy: Deep generative modeling of nonlincar stochastic governing equations. arXiv preprint arXiv:2310 04832, 2023.
S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti, and Kumar. Rethinking FID: Towards a better ovaluation metric for image generation. IEEE Conference on Computer Vision and Paltern Recognation (CVPR), 2023.
22

---

Published in Transactions on Machine Learning Research (08/2024)
P. Jeha, M. Bohlkk-Schneider, P. Mercado, S. Kapoor, R. S. Nirwan, V. Flunkert, J. Gasthaus, and T. Januschowski. PSA-GAN: Progressive self attention GANs for synthetic time series. Tn International Conference on Learning Representations, 2021.
W. Jitkrittum, 7. Szabó, K. P. Chwialkowski, and A. Gretton. Interpretable distribution features with maximum testing power, In Advances in Neural Information Processing Systems, 2016.
J. Kapoor, A. Schulz, J. Vetter, Pei, R. Gao, and H. Macke. Latent diffusion for neural spiking data. arXiv preprint arXit:2407.0875, 2024.
I. Kim, A. B. Lee, and J. Lei. Global and local two-sample tests via regression. Electron. J. Statist., 2019.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
S. Kolouri, K. Nadjahi, U. Simsekli, R. Badeau, and G. Rohde. Generalized Sliced- Wasserstein distances. Advances in Neural Information Processing Systems, 2019.
J. M. Kübler, W. Jitkrittum, B. Schölkopf, and K. Muandet. A witness two-sample test. In International Conference on Artificial Intelligence and Statistics, 2022a.
J. M. Kübler, V. Stimper, S. Buchholz, K. Muandet, and B. Schölkopf. Automl two-sample test. In Advances in Neural Information Processing Systems, 2022b.
H. W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 1955.
T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, and J. Lehtinen. The role of imagenet classes in Fréchct inception distance. International Conference on Learning Representations, 2023.
T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. Conference on Neural Information Processing Systems, 2019.
C. Leslie, E. Eskin, and W. S. Noble. The spectrum kernel: A string kernel for svm protein classification. In Biocomputing, 2001.
C.-L. Li, W.-C, Chang, Y. Cheng, Y. Yang, and B. Póczos. MMD-GAN: Towards deeper understanding of moment matching network. Advances in Neural Information Processing Systems, 2017.
Y. Li, K. Swersky, and R. Zemel. Generative moment matching networks. International Conference on Machine Learning, 2015.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Rision-ECCU 2014: 13th European Conference, Zurich, Swilzerland, September 6-12, 2014, Proceedings, Parl V 13, 2014.
F. Liu, W. Xu, J. Lu, G. Zhang, A. Gretton, and D. J. Sutherland. Learning deep kernels for non- non- parametric two-sample tests. In Proceedings of the 37th International Conference on Machine Learning, 2020.
L. Liu, K. Pillutla, S. Welleck, S. Oh, Y. Choi, and Z. Harchaoui. Divergence frontiers for generative models: Sample complexity, quantization effects, and frontier integrals. Aduances in Neural Information Processing Systems, 2021.
A. Liutkus, U. Şimgekli, S. Majewski, A. Durmus, and F.-R. Stöter. Slisee-Wasserstein flows: Nonparametric generative modeling via optimal transport and diffusions. International Conference Machine Learning, 2019.
J. R. Lloyd and Z. Ghahramani. Statistical model criticism using kernel two sample tests. In Advances in Neural Information Processing Systems, 2015.
23

---

Published in Transactions on Machine Learning Research (08/2024)
H. Lodhi, C. Saunders, J. Shawe- Taylor, N. Cristianini, and C. Watkins. Text classification using string kernels. Journal of Machine Learning Research, 2002.
D. Lopez-Paz and M. Oquab. Revisiting classifier two-sample tests. International Conference on Learning Representations, 2017.
J.-M. Lueckmann, J. Boelts, D. Greenberg, Goncalves, and J. Macke. Benchmarking ioauulation-bbsee inference. In International Conference on Artificial Intellignnce and Statistics, 2021.
A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. V. Gool. RePaint: Inpainting using denoising diffusion probabílistic models. Proceedings of the SEEE/CVE conference on computer vision and pallern recognilion, 2022.
M. Malik and D. Humair. Evaluating the fcasibility of using generative models to generate chest X-ray data. Xrviv preprinl 2r0iv:2305. 18927, 2023.
M. Marouf, P. Machart, V. Bansal, C. Kilian, D. S. Magruder, C. F. Krebs, and S. Bonn. Realistic in silico generation and augmentation of single-cell rna-seq data using generative adversarial networks. Nature communications, 2020.
A. Mathiasen and F. Hvilshøj. Backpropagating through fréchet inception distance. ar preprint W:Xiv:2009114075 2021.
Midjourney. Midjourney, 2022. URL http://wwwwiifoorrreccoo//o///ee Accessed: 2024-06-06.
M. Molano-Mazon, A. Onken, E. Piasini, and S. Panzeri. Synthesizing realistic neural population activity patterns using generative adversarial networks. International Conference on Learning Representations, 2018.
K. Muandet, K. Fukumizu, B. Sriperumbudur, and B. Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning, 2017.
M. Müller. Dynamic time warping. Information retricval for music and motion, 2007.
K. Nadjahi. Sliced Wasserstein distance for large-seale machine learning: theory, methodology and eztensions. PhD thesis, Institut polytechnique de Paris, 2021.
K. Nadjahi, A. Durmus, L. Chizat, S. Kolouri, S. Shahrampour, and U. Simsekli. Statistical and topological properties sliced probability divergences. Advances in Neural Information Processing Systems, 2020.
K. Nguyen and N. Ho. Sliced wasserstein estimation with control variates. International Conference Learning Representations, 2024.
A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. International Conference Machine Learning, 2022.
v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.022 2016.
M. Pals, A. E. Sağtckin, Poi, M. Glocckler, and J. H. Macke. Inferring stochastic low-rank recurrent neural networks from neural data. arXiv preprint arXiv:2406. 16749, 2024.
V. M. Panarctos and Y. Zemel. Statistical aspects of Wassorstein distances. Annual review of statistics and ils application, 2019.
T. Pandeva, T. Bakker, C. A. Nacsseth, and P. Forré. E-valuating classifier two-sample tests. Transactions on Machine Learning Research, 2024.
G. Papamakarios, E. Nalismick, D. J. Rezende, S. Mohamed, and B. Iakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 2021.
24

---

Published in Transactions on Machine Learning Research (08/2024)
T. Papp and C. Sherlock. Bounds wasserstein distances between continuous distributions using independent samples. Xiv preprint ar arx:c:200311 1627, 2022.
G. Parmar, R. Zhang, and J.-Y. Zhu. On aliased resizing and surprising subtleties in GAN evaluation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2022.
F. Podregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Rescarch, 2011.
G. Peyré, M. Cuturi, et al. Computational optimal transport. Center for Research in Economics and Stafistics Working Popers, 2017.
J. W. Pillow, J. Shlens, L. Paninski, A. Sher, A. M. Litke, E. Chichilnisky, and E. Simoncelli. Spatio- temporal correlations and visual signalling in a complete neuronal population. Nature, 2008.
K. Pillutla, S. Swayamdipta, R. Zellers, J. Thickstun, S. Welleck, Y. Choi, and Z. Harchaoui. Mauve: Mcasuring the gap between neural text and human text using divergence frontiers. Advances in Newral Information Processing Systems, 2021.
K. Pillutla, L. Liu, J. Thickstun, S. Welleck, S. Swayamdipla, R. Zellers, S. Oh, Y. Choi, and Z. Harchaoui. Mauve scores for generative models: Theory and practice. Journal of Machine Learning Research, 2023.
G. Pombo, R. Gray, M. Cardoso, S. Ourselin, G. Rees, J. Ashburner, and P. Nachev. Equitable modelling of brain imaging by counterfactual augmentation with morphologically constrained 3d deep generative models. Medical Image Anal. 2021. doi: 10.1016/j 10.1016)).mciia22022.. modia 202222
A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training, OpenAl blog, 2018.
A. Radford, J. WU, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
A. Radford, J. W. Kim, C. Hallacy A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, 2021.
S. Raschka. An overview of general performance metrics of binary classifier systems. arXiv preprint arXiv:1410.5880 2014.
R. Ratcliff. A theory of memory retrieval. Psychological review, 1978.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st. International Conference on Machine Learning, 2014.
J. Roitman and M. Shadlen. Response of neurons in the lateral intraparietal area during combined visual discrimination reaction time task. Journal of Neuroscience, 2002.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. fiighrreoluutin image synthesis with latent diffusion models. In Proceedings of the IEFF/CVF conference computer vision and pattern recognition, 2022a.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022b.
C. Rommel, J. Paillard, T. Moreau, and A. Gramfort. Data augmentation for learning predictive models EEG: systematic comparison. journal of Neural Engineering, 2022.
M. M. Saad, R. O'Reilly and M. H. Rehmani. A survey on training challenges in generative adversarial networks for biomedical image analysis. Artificial Intelligence Review, 2024.
25

---

Published in Transactions on Machine Learning Research (08/2024)
Т. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training GANs. Conference on Neural Information Processing Systems, 2016.
A. Schrab, I. Kim, M. Albert, B. Laurent, B. Guedj, and A. Gretton. MMD aggregated two-sample lest. Journal of Machine Learning Research, 2023.
C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 2022.
A. Schulz, J. Vetter, R. Gao, D. Morales, V. Lobato-Rios, P. Ramdya, P. J. Gonçalves, and H. Macke. Modeling conditional distributions of neural and behavioral data with masked variational autoeneoders. bioRrin, 2024.
B. Segal, D. Rubin, G. Rubin, and A. Pantanowitz. Evaluating the clinical realism of synthetic chest X-rays generated using progressively growing GANs. SN Computer Science, 2021.
D. Sejdinovic, B. Sriperumbudur, A. Gretton, and K. Fukumizu. Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The Annals of Statislies, 2013.
M. Shinn, N. H. Lam, and J. D. Murray. A flexible framework for simulating and fitting generalized drift-diffusion models. ELife, 2020.
K. E. Smith and A. O. Smith. Conditional GAN for timeseries generation. arXiv preprint or Xix:2200 16477, 2020.
R. S. Somerville and R. Davé. Physical models of galaxy formation in cosmological framework. Annual Review of Astronomy and Astrophysics, 2015.
Y. Song, Sohk-Dickstein, D. Kingma, A. Kumar, S. Ermon, and B. Poole. Score- based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Y. Song, P. Dhariwal, M. Chen, and Sutskever. Consistency models. arxis preprint arXiw:2903.01402 2023.
B. K. Sriperumbudur, K. Fukumizu, A. Gretton, G. Lanckriet, B. Schölkopf, and B. K. Sriperumbudur. Kernel choice and classifiability for RKHS embeddings of probability distributions. Advances in Neural Information Processing Systems, 2009.
B. K. Sriperumbudur, K. Fukumizu, and G. R. Lanckrict. Universality, characteristic kernels and RKHS embedding of measures. Journul of Machine Learning Research, 2011.
A. Srivastava, K. Xu, M. U. Gutmann, and C. Sutton. Generative ratio matching networks. International Conference on Learning Representations, 2020.
C. Stringer, M. Pachitariu, N. Steinmetz, C. B. Reddy, M. Carandini, and K. D. Harris. Spontaneous behaviors drive multidimensional, brainwide activity. Science, 2019.
M. Sugiyama, T. Suzuki, and T. Kanamori. Density Ratio Estimation in Machine Learning. Cambridge University Press, 1st edition, 2012. ISBN 0521190177.
D. J. Sutherland. Maximum mean discrepancy (distance distribution). Cross Validated, 2019. URL
D. J. Sutherland, H.-Y.
Tung, H. Strathmann, S. De, A. Ramdas, A. Smola, and A. Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In International Conference on Learning Representations, 2021.
C. Szegody, W. Liu, Y. Jia, P. Sermanct, S. Reod, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognation (CVPR), 2015.
26

---

Published in Transactions on Machine Learning Research (08/2024)
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. IEEF Conference on Computer Vision and Pattern Recognation (CVPR), 2016.
G. J. Suékely and M. L. Rizzo. Energy statistics: A class of statistics based on distances. Journal of Stalistionl Planming and Inference, 2013.
A. Vejero-Cantedd J. Boelts, M. Deistler, J.-M. Lueckmann, C. Durkan, P. J. Gonçalves, D. S. Greenberg, and J. H. Macke. sbi: A toolkit for simulation-based inference. journal of Open Source Software, 2020.
L. Theis, A. van den Oord, and M. Bethge. A note on the evaluation of generative models. International Conference on Learning Representation, 2016.
M. K. Titsias and F. Ruiz. Unbiased implicit variational inference. Tn Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, 2019.
E. Tiu, E. Talius, P. Patel, C. P. Langlotz, A. Y. Ng, and P. Rajpurkar. Expert-level detection of pathologies from unanotated chest X-ray images via self-supervised learning, Nature Biomedical Engineering, 2022.
F. Urbina, C. T. Lowden, J. C. Culberson, and S. Ekins. MegaSyn: integrating generative molecular design, automated analog designer, and synthetic viability prediction. ACS omega, 2022.
C. Vallez, A. Kucharavy, and L. Dolamic. Needle in haystack, fast:: Benchmarking image perceptual similarity metrics at scale. arXiv preprint arXiv:2206. 200822, 2022.
A. van den Oord, N. Kalchbrenner, L. Espeholt, K. Kavukcuoglu, O. Vinyals, and A. Graves. Conditional image generation with pixelenn decoders. In Advances in Neural Information Processing Systems, 2016.
J. Vetter, J. H. Macke, and R. Gao. Generating realistic neurophysiological time series with denoising diffusion probabilistic models. bioRaie, 2023.
J. Vetter, G. Moss, C. Schröder, R. Gao, and J. H. Macke. Sourcerer: Sample-based maximum entropy source distribution estimation. arXie preprint arXiv:2402 07808, 2024.
S. V. N. Vishwanathan, N. N. Schraudolph, R. Kondor, and K. M. Borgwardt. Graph kernels. Journal of Machine Learning Research, 2010.
X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. Summers. X- Hospital-scale chest X-ray database and benchmarks on suakki--upervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
J. Wu, Z. Huang, D. Acharya, W. Li, J. Thoma, D. P. Paudel, and L. V. Gool. Sliced wasserstein generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), June 2019.
Wukong. Wukong, 2022. URL https://xihe www.mmiiinndnni.ooomm cn/model2oo//uukogg Accessed: 2024-06-06.
Z. Xiao, K. Kreis, and A. Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. International Conference on Learning Representations, 2022.
Q Xu, G. Huang, Y. Yuan, C. Guo, Y. Sun, F. Wu, and K. Weinberger. An empirical study on evaluation metrics of generative adversarial networks. arXie preprint arXiv:1806.07755, 2018.
C. Yang, Y. Zhang, Q. Bai, Y. Shen, B. Dai, et al. Revisiting the evaluation of image synthesis with gans. Advances in Neural Information Processing Systems, 2023.
G. Yenduri, R. M, C. G, S. Y, G. Srivastava, P. K. R. Maddikunta, D. R. G, R. H. Jhaveri, P. B, W. Wang, A. V. Vasilakos, and T. R. Gadckallu. Generative pro-trained transformer: A comprchensive review on enabling technologies, potential applications, emerging challenges, and future directions. IEEE Access, 2024.
27

---

Published in Transactions on Machine Learning Research (08/2024)
M. Yi and S. Liu. Sliced- Wasserstein variational inference. In Proceedings of The 14th Asian Conference Machine Learning, 2023.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the TEFE conference computer vision and pattern recognition, 2018.
J. Zhao and D. Meng. FastMMD: Ensemble of circular discrepancy for efficient two-sample test. Neural Computation, 2015.
M. Zhu, H. Chen, Q. Yan, X. Huang, G. Lin, W. Li, Z. Tu, H. Hu, J. Hu, and Y. Wang. Genimage: A million-scale benchmark for detecting ai-generated image. Conference on Newral Information Processing Systems, Track on Datasets and Benchmorks, 2023.
28

---

Published in Transactions on Machine Learning Research (08/2024)
A Appendix
A.1 Generative Models in Science
Table S1: Example generative models in science. This is not an exhaustive list regarding disciplines using generative models nor generative models used in the listed disciplines.
Discipline
Model
Biology
Neuroscience
1
single cell sequencing
cellular biology 25
26
Geoscience
ice flow modelling
00
Numerical weather prediction
Chemistry
molecule generation +
48
Astronomy
astronomical images
59 63
A.2 Details about Maximum Mean Discrepancy
Here we provide different formulations and examples MMD.
Definition A.1 (Feature Map Definition of MMD)
MMD3 Pl, Pt] = (x) (x) [dv)]
(5)
where (I()) and p2(y) are the probability distributions of random variables z, X, and Φ X H.
Generally, X and H are defined a topological space, and the reproducing kernel Hilbert space (RKHS), respectively, but readers can simply think of the Euclidean space N for the first examples in the main text.
For the identity feature map 6()) R R, (x) = x, MMD can be computed as
MMD (()(L)
-21,22] = (() (z) [xc] Ept(v) w
= =(Ep1 (w) [x]]
Epa [x))
MMD (1) -P1,P2] 11pp 1ppll-
And for the guadvatic polynomial fenture map ((2)
R R.2 (xx)
x
MMD can be computed as
2
MMD ,PI,P2] = | [p)
x
| -EE Epp(
x2
y2
= 16 + o
up
Mpz
11
Ub + 2
p2.
MMD2 P1.P2] (Mp. Mpp)
+ (u) + op
o2, ya
Definition A.2 (Kernel Definition of MMD)
MMD MMD2(0.PP.PP P1. = [p((()))(()) (p))))
(o)))) Ep, (o)x))))))]]] + (B2 Bmm. 2(E 2
Epin) (z)) ((1))) Emain ((d())) ((b()))
() - (o (())))
MMD² [k, PP) (p. ( v/)) (k(
29

---

Published in Transactions on Machine Learning Research (08/2024)
The definition of MMD can be rewritten through the notion kernel meon embedding. For given distribution p(æ), the kernel mean embedding Bp(c) (I, H is an element in RKHS that satisfies (By(s) (M(,, [(u))x (f(z))(())) for any f E H. with argument u E X. The embedding Ep(a) [k(,, u)] is known to be determined uniquely if a corresponding kernel is bounded, i.e., [|A(r, I') < 2o for any æ. Then, as shown in Gretton et al. (012a), MMD* can be represented as
MMD Pr. P2] = (p) [k(x, u)] - Po [k(y: u)]l
MMD can also be defined more generally as the integral probability metric.
Definition A.3 (Supremum Definition of MMD)
MMD[F ..DDFFPP.P0 P1. fer
sup
(6)
Here, F is a class of functions f: X R. Where we take F as the unit ball in an RKHS H with associated kernel k(x,x') (Grelton et al. 2012aa, the function that atlains supremum (the wilness function) is
Ep (11 w)] Ep)( [k w)] (m) (k( - Eps [k 2) l)]l
f(u)
Assigning f(u) into (Eq. 6 ) we have
MMD Pp] = Sup (B
pi (m)) [f(x')] - -E2o))((/))1)
JEF
E (1) (kx,, x')l Ep [uy, /)1 (en( 2
[p(() [xx w]]
Bp(()) (VV, w)]l
(0) (k(r, u)] (x) [N u) u)]l 11
[Bp [M<(( u)] (k) (ky, u)]l
ppp [k(2, w)] pp [k(y, m)ll
which is equal to the kernel definition of MMD.
Definition A.4 (Characteristic Kernel) A kernel is called characteristic when the kernel mean embedding
p(z) fu) Ep(x) u)] e: H.
is injective (Sriperumbudur et al.
2011 Fukumizu et al. 2008).
This means that, if a characteristic kernel is used, the embedding into the RKHS can uniquely preserve all information about a distribution. In our evaluation, we utilize the Gaussian kernel, one of the well-known characteristic kernels. Another example of characteristic kernel is the Laplacian kernel, which is defined by
k(x,x') == exp(cxppp -p00-Bla- z'l).
Note that linear and polynomial kernels are not characteristic, while they are quite popular in natural language processing.
A.3 Formal Definition of Wasserstein and Sliced- Wasserstenn Distance
Let pM, be Polish space, n E P(M) be two probability measures, and let q [1, +00)) Then the Wasserstein-g distance between μ and v is defined
W,(4v v) = ( inf ()E((H.))
(r) 1/9 ((),))))(
30

---

Published in Transactions on Machine Learning Research (08/2024)
10-dim Gaussian
100-dim Gaussian
1000-dim Gaussian
0.3
0.1
0.04
SW
SW
true
SW
true
shifted
true
shifted
shifted
0.0
0.0
0.00
50
100
150
200
50
100
150
200
0
50
100
150
200
projections
projections
projections
Figure S1: The SW distance estimate is not strongly sensitive to the number of random projections. We compare the SW distance cstimate for the {10, 100, 1000)-dimensional Gaussian task with 1 shifted dimension (Sec. 3 as we increase the number of random projections used in the estimation. As the number of projections increases, the variance of the SW distance estimate decreases, but across all dimensionalities considered, the SW distance estimate has converged by 100 random projections.
where D) is the of all couplings between J.
The Sliced- Wasserstein distance is closely connected to the Radon transform (Helgason 1980 We direct readers to Bonneel et al. (2015) for details. Here provide a shorter definition following Definition 2.9 of Nadjahi 20011.
Suppose that M C Rd, and denote by gd-1 = {0 E TRd ||01|2 = 1} be the unit sphere with respect to the Euclidean norm. Let u* X + R be a the linear form given by u wx(x) = (u, and q E [1, +oc). Then the Wasscrstsnn distance (of order q) definod for any measures p, v E Pg(X) as
SW = (1 Wi )
1/9
Where U is the uniform distribution Sd-1, and for any gd-1 u denotes the push-forward operator of u².
As for the Wasserstein distance, the definition becomes more intuitive in the case where μ and v admit the probability density functions (pi and pa respectively). In particular, the random projection u are uniformly random vectors in gd-1, Therefore, projecting the distributions p1 and p2 onto u induces one-dimensional distributions pf and pz with samples u x,, where 21 ~ P1 and 22 ~ P2. The Sliced- Broood-Wasersttinngg distance can then be written as
SW,(P1,P) P2) (W 1/9
A.4 Dependence of SW Distance on Number of Projections
All SW distance experiments in the main text were performed with the SW distance approximate with 100 random projections to approximate the expectation Eq. (7) Here, we additionally show the dependence of the SW distance approxination with finite projections the d-dim Gaussian example (see Sec. 3 for de {10, 100, 1000} (Fig ST While the sample-based approximation to the analytic 1-dimensional Wasserstein distance is biased (Sec. 2.1 the Monte Carlo approximation to the expectation Eq. (7) is an unbiased estimate of the sample- base- Wasserstein distance. We observe that for very high (1000)-dimensional distributions, the SW distance estimate converges quickly, and the choice of 100 random projections for the computation of the SW distance is appropriate.
A.5 C2ST scores below 0.5
In practice the C2ST score can sometimes turn out be below .5. That is, the trained classifier performs systematically worse than random classifier. A potential reason for this effect the existence of near duplicates copies between the two different sets. Before training the classifier, these duplicates assigned
31

---

Published in Transactions on Machine Learning Research (08/2024)
Table S2: Summary of computational complexity and theoretical properties of metrics in terms
of number of samples N and data dimensionality D. Sample complexity here refers to the convergence
rate of the sample-based estimate to the true value of the metric. *Bound based on Ghosal & Sen (011);; Nguyen & Ho (2024) for SWD and Gretton et al. (2012a) for MMD. 'Best case scenario. Tn practice,
the computational cost of training neural network scales superlinearly with both sample size and data
dimensionality. I General case (scc Sec. 2.3
A.6 for details)." Cost of calculating the square root of the
covariance malrix in Eq.
SW
C2ST
MMD
FID
Sample Complexity (N)
O(N 1/2)=
N/A
((N-1/2)*
N/A
Computational Complexity (N)
ON log N)
O(N)
O(N3)
O(N2)
Computational Complexity (D)
((D)
O(D)
O(D)
O(D
Estimator unbiased?
no
no yes
no
opposite class labels. When a given pair of such duplicates is then split into one that belongs to the training set and one that belongs to the test set, the classifier is biased towards predicting the wrong class for the duplicate in the test set. This effect is particularly noticeable the classifier was not carefully regularized during training, and thus memorized the class label of the duplicate in the training set.
A.6 Sample and computational complexity
We here highlight that the different distances also differ in their sample and computational complexities (Table S2 With respect to the number of samples N, MMD and FID have complexity of O(N2). Note
that for MMD the computational cost can be reduced, e.g., by increasing variance or for a specific kernel
choice (rretoon et al. 2012a Zhao & Meng 2015: Cheng & Xie 2021 Bodenham & Kawahara 2023 Bharti
iet al. 2023 Gretton et al. (012b]). While SW distance scales with O(N log (Nadjahi 2021 it is dificult
to make principled asscssments of the computational complexity for C2ST, as is highly dependent on the chosen classifier. However, as more samples lead to larger training and test datasets, the sample size is likely to influence the compute time. Similarly, the computational complexity of computing these distances increases as the dimensionality of the data increases, with non-trivial scaling depending on the task and chosen hyperparameter. We also report the theoretical convergence of sample- based estimates for MMD, as well as SW, which is subject to active research. We report bounds from recent works (Ghosal & Sen 2019: Gretton et al. 2012a). We do not report sample complexitics for C2ST and FID, as these strongly depend on the choice of classifier and embedding nerkork, respectively.
Note that despite their differences, all presented distances are reasonably tractable in the settings of our experiments, whereas the scaling experiments might be computationally unfeasible for other distances datasets. Therefore, we strongly recommend carefully considering the complexity of the measure before conducting experiments on high-dimensional or very large datasets. We report the practical computation times for our experiments in Appendix A.1l
A.7 Mode coverage properties
Mode coverage is the ability of a model to capture and generate diverse data, i.e. from multiple modes of the underlying distribution (sce Fig. S2 S3 The community has focused on evaluation of mode coverage with different metrics driven by the development of GANs (Goodfellow et al., 2014 Gui et al., 2023 Saad et al.) 2024). Empirically, in training generative models, mode coverage has been found to trade off with sample quality and speed, illustrating the generative learning trilemmaa (Xiao et al. 2022 All presented metrics can distinguish between a collapsed and a full distribution (Fig. 9 an empirical finding also reported in previous work (Che et al.) 2017: Li et al. 2017 Deshpande el al.] 2018 Borji 2019). However, their sensitivity relies on different factors: SWD captures different modes when they are separated in one-dimensional projections, MMD depends on appropriate kernel choice, FID on expressive embeddings, and C2ST on well-trained classifiers. Other metries used to quantify mode collapse include precision and recall (Kynkäänniemi et al.
2019
32

---

Published in Transactions on Machine Learning Research (08/2024)
Pinee
SW
C2ST
MMD,
Figure S2: Trade-offs illustrated through optimization of miss-specifled model. We fitted misspecified model (a two-dimensional Gaussian) by using different distances to multi-modal distribution (similar to Theis et al. 2016). Note, that for well-specified model each distance would give a perfect fit (Section A.8.2). 1. In the optimization we minimized the SW distance, the C2ST classifiability, and the MMD with Gaussian Kernel (details in Section A.8.1). Plotted are the contour lines of .2, .75, 1, and 2.5 standard deviations of the fitted Gaussians. The model optimised with SW mass-covering: it covers both modes and therefore also assigus density to low-density regions of the true distribution, thus producing varied, but. potentially unlikely samples. The models optimised with C2ST and MMD are mode-seeking: they have high densitics only in the largest mode of the true distribution, and thus produces likely, but unvaricd samples.
A.8 Sliced- Wasserseis, MMD and C2ST optimization target
A.8.1 Fitting a Gaussian with gradient descent
We provide an illustrative example of which distributions are obtained when using Wasserstein, MMD and C2ST distances as a goodness of fit. criterion, for both a miss-specified example, in Fig. S2 and well-specified example Fig. S3 We can see that in the miss-speciffed example different distances make different trade-offs, for example whether they are mode-seeking, and produce likely but unvaried samples, or are mode-covering, where they produce varied, but also potentially unlikely samples.
For Wasserstein, optimisation has been studied more formally in previous work Bernton et al. 2019
& Liu 2023). Wasserstein was used training objective in Arjovsky et al. (2017) and MMD was used training objective in Binkkowki et al. (2018): Dziugaite et al. (2015); Li et al. (2015). Optimizing the C2ST classifier at the same time as the parameters of our generative model is similar to training GAN (Goodfellow
et al. 2014), but for simplicity we instead optimized for the closed-form optimal C2ST as for this toy example we have access to true densities. While FID can be used as optimizatiin target in principle (Mathiasen & Hvilshoj 2021), its applicability our toy example here is less obvious, so we excluded here.
In order to fit the (miss-specified) Gaussian model,
p(x) = N(,,
to the ground truth distribution Pirue which is a mixture of Gaussians, we proceed as follows. Let CCT denote the Cholesky decomposition of Σ. We compute gradients with respect to μ and C by using the reparameterization trick; by generating samples as + Ce, with ~~ I).
For Wasserstein we used loss the Sliced Wasserstein distance, for MMD, used Gaussian Kernel with bandwidth set according the median heuristic.
For C2ST, we can evaluate the probability densities of samples from both the learned Gaussian and the ground truth mixture of Gaussians, so we minimize the accuracy of the closed-form optimal classifier. For each sample, we evaluate the log- bog-probabylity density of the sample under each distribution, softmax the two resulting values, and use those as the classifier predicted probabílities. We then use binary cross-entropy the loss function.
33

---

Published in Transactions on Machine Learning Research (08/2024)
We used the ADAM optimizer (Kingma & Ba 2015 with learning rate=0.01 and default momentums, using 2500 epochs of 10000 samples.
A.8.2 Fitting a mixture of Gaussians with Expectation-Maxiiisttioo
We also include an example where the model we fit is well-specified, which in this case means it also a mixture of two Ganssians (Fig. SS As directly optimizing mixture distribution with gradient descent is not straightforward, we used an Expectation-Mel Mexiiiiionnaannnizatiinnn algorithm (where the distances instead of the log-likclihood in the maximisation step)
Our model is specified by
p(x) (Hn Σ1) (22 Σz)
which can write latent-variable model, where the latent variables are the cluster assignments:
p(z) Σ p(a|= k)p(z k),
k=1
with p(x|2 1:) NWu, Σx) and p(= k3) wk.
We then iteratively performed the following two steps to optimise the model.
E-step: For each of the N datapoints 2; from Purue, we calculated the probability of it belonging to mixture component 1 or 2:
p(2 k(a,) p(0,12 p(2,1==1)ppe k)p(2 1))
p(al=== k')p(= k')
M-step: We updated the mixture weights according to:
1 W/k N Σ p(== = k(2.)
Next, for each of the N datapoints, we first sampled a cluster assignment according to 2; ~ p(=[2.). Then for each group of N& datapoints assigned cluster l we sampled N& times according to x; ~ p(x]=)), again using the reparameterisation trick. As before computed the loss using statistical distance, now separately for the two groups of samples assigned to either mixture component, and used gradient to optimise μ4, Ex
Again, we used the ADAM optimizer using 2000 epochs of 5000 samples.
Kingma & Ba. 2015). with learning rate=0.01 and default momentums,
34

---

Published in Transactions on Machine Learning Research (08/2024)
Pinu
SW
C2ST
MMD,
O
Figure S3: Optimising distances in well-specified model setting When fitting well-specified model (here, mixture of two Gaussians), by using different distances in the loss, see that each model converges to the global optimum. Plotted are the contour lines of .255 .75, 1, and 2.5 standard deviations of the fitted Gaussians multiplied by their corresponding mixture weight.
A.9 Additional scaling experiments with different sample size budgets and ranges
In Fig. g evaluated the robustness of the measures against the number of samples and the dimensionality of the data. We observed notably poor performance of the measures in scenarios with limited data. Here, we further examine the performance of the distances across datasets of varying sample sizes, particularly for small sample set sizes, ranging from only 8 to 80 samples per set (Fig. [54). We examine three distinct data configurations where the distinction between the true and approximated distributions progressively increases from subpanels [S4 a c. Across all distances, becomes evident that the larger the disparity between the two distributions, the fewer samples are needed for differentiation. In the experiment where all dimensions are mean-shifted by one, sample size of is sufficient to distinguish between the distributions. However, for less distinct distributions, such as the unimodal Gaussian mean-shift by one in only one dimension (Supp. Fig. S4 a, b), all distances exhibit poor performance in distinguishing between the distributions.
35

---

Published in Transactions on Machine Learning Research (08/2024)
b
2-dim Mixture of Gaussians vs. unimodal Gaussian
10-dim Gaussian
10-dim Gaussian
vs. first dimension shifted by
vs. dimensions shifted by 1
true
true
true
approx.
0.75
shifted
shifted
0.50
0.5
0.25
1.0
true
true shifted
C2ST
врprox.
C2ST
C2ST
true
shifted
0.5
0.5
0.5
0.3
true shifted
0.4
true
MMD
0.2
approx.
MMD
MMD
shifted
0.2
0.1
0.0
50
25
50
75
sample size
sample size
sample size
Figure S4: The larger the difference between two distributions the fewer samples suffice to tell the true and shifted distribution apart. We compare sample sets with varying sample sizes (between 8 and 80 samples per set) of 'true' distribution either with second dataset of the same distribution with sample set from an approximated/shhite /shiftee distribution. We show the mean and standard deviation over five runs of randomly sampled data. (a) Distances for the 2d-MoG example shown in Fig. compared to samples from a unimodal Gaussian approximation with the same mean and covariance. (b) Distances for a ten-dimensional standard normal distribution, for which the first dimension is shifted by one for the shifted example. (c) Distances for a ten-dimensional standard normal distribution, for which all dimensions are shifted by one for the shifted example.
A.10 Additional scaling experiments for different dimensionality of the data
When comparing the robustncss of the mcasures with respect to the dimensionality of the data in Fig. we observed a degradation in the ability to distinguish between distributions as dimensionalities increased. Notably, only the C2ST measure retained the capability to distinguish between the two distributions in higher dimensions which is aligned with the intuition that classifier can easily pick up on differences in a single dimension. Extending this analysis, Fig. S5 presents similar experiments conducted compare an n-dimensional standard normal distribution with one where either all dimensions are mean-shifled by one (thus aligning with the C2ST experiment in Fig. 5p) or where all variances are increased by one. Fig. [55k corresponds to the experiment outlined in Fig. Sp on dimensionality. The bandwidth parameter for the MMD distance has been adjusted to suit the particular data configuration and is represented by the integer in the y-axis label. Generally, we notice that the Blicod-Wassersteii distance and MMD face difficulties in higher-dimensionall spaces, especially when handling distributions that are only slightly distinct if the respective hyperparameters kept constant across dimensions. In contrast, the C2ST dis- tance consistently demonstrates good performance across all three experiments and for all ranges of dimensions.
datasets where we
A.11 Comparisons practical compute times of different measures
Before computing such measures, in particular for scaling experiments such as the ones presented here, where measures calculated across large range of sample sizes N and data dimensions D, the practical compute
36

---

Published in Transactions on Machine Learning Research (08/2024)
n-dim Gaussian
b
n-dim Gaussian
n-dim Gaussian
first dimension shifted by
all dimensions shifted by 1
vs. all variances increased by
0.4
0.4
true
shifted
increased
SW
SW
0.2
0.5
true
0.2
shifted
1.0
1.0
true shifted
C2ST
C2ST
true shifted
C2ST
true increased var
0.5
0.5
0.5
0.010
0.3
true shifted
true
MMD10
MMD10
0.2
MMD10
Increased var
0.005
0.1
0.02
shifted
0.000
0.0
0.00
10
10
10
10
10"
10
10
10
10"
dimensions
dimensions
dimensions
Figure S5: The impact of dimensionality varies across distances, with certain distances facing particular challenges in higher dimensional spaces. We compare sample sets of varying dimensionality (between 5 and 1000) of a 'truc' distribution either with a second dataset of the same distribution or with a sample set from an approximated/shiitel distribution. The sample size is fixed to 10k for all experiments and we show the mean and standard deviation over five runs of randomly sampled data. The bandwidth parameter Gaussian Kernel MMD is set to 10 for all experiments.
(a) Distances for sample set from an n-dimensional standard normal distribution, for which the first dimension shifted by onc. (b) Distances for sample set from an n-dimensional standard normal distribution, for which all dimensions shifted by one. (c) Distances for sample set from an n-dimensional standard normal distribution, for which variances are increased by one for all dimensions.
time of the chosen measures should be considered. Depending on the downstream application, might be time-critical to quickly evaluate distances which might favor some measures over others. Aligned with theoretical considerations regarding sampling complexity etc. as presented in Table $2), empirical compute times vary between the different measures. Given that empirical computational times for single measure itself vary depending on the exact implementation, compute infrastructure, and problem at hand, we list approximated compute times for running the scaling experiments in Table S33 The calculated runtime combines both the comparisons of the 'truc' and the 'shifted' or 'approximated" experiments. Each experiment contains five repeats across different sampled data subsets. The sample size experiment contains eight different. sample size values N (50, 100, 200, 500, 1000, 2000, 3000, 4000), and the dimensionality experiment scales tests six different D (5, 10, 50, 100, 500, 1000). For more details, see Section 3 The version of C2ST we use here which is based on Na-based classifiers, takes orders of magnitude longer to compute than SW and MMD. Note, however, that alternative implementations and classifier variants could speed this up.
37

---

Published in Transactions on Machine Learning Research (08/2024)
Table S3: CPU wallclock run times for the comparison scaling experiments in Fig
The runtime combines
both the comparisons of the true the shifted or approximated distribution. Values are rounded estimates.
sample size experiment
dimensionality experiment
2-dim MoGs
10-dim Gaussian
n-dim Gaussian
SWD
0.3 8
0.3 s
1.5
C2ST
150 8
300 s
1500
MMD
3
3 s
80 s
A.12 Sensitivity of the MMD kernel choices and hyperparameters
The general formulation of MMD allows for wide range of kernel choices, cach potentially with their own
hyperparameters. These choices can have significant effect on its behavior. We provide some experiments
demonstrating of the importance of well-tuned bandwidth parameters for Gaussian Kernel MMD as well as the impact different kernels.
a
2-dim Mixture of Gaussians
b
10-dim Gaussian
100-dim Gaussian
true
true shifted
true shifted
MMD
0.01
approx.
MMD
0.02
0.002
0.00
0.00
0.000
bandwidth
bandwidth
bandwidth
Figure S6: The bandwidth parameter in Gaussian Kernel MMD is sensitive parameter that requires careful selection for each dataset. The sample size is fixed to 10k for all experiments and we show the mcan and standard deviation over five runs of randomly sampled data. (a) MMD² distance with varying bandwidth parameters between 0.1 and 5 for the 2d-MoG example compared to samples from a unimodal Gaussian approximation with the same mean and covariance. (b) MMD² distance with varying bandwidth parameters between 0.1 and 20 for a 10-dimensional standard normal distribution, for which the first dimension is shifted by one for the shifted example. (c) MMD distance with varying bandwidth parameters bet ween 1 and 40 for 100-dimensional standard normal distribution, for which the first dimension is shifted by one for the shifted example.
We first vary the bandwidth parameter with fixed sample sizes for the three example datasets used in Section (Fig, S6). We show that the estimated MMD values vary significantly across bandwidths, and both setting the bandwidth too low or too high yield poor results. However, we note that the values yielded by the median heuristic (bandwidths of 1, 5, and 10 for the three datasets, respectively, as shown the main text) are quite near the peaks of the curves at which MMD most effectively distinguishes the distributions.
We then choose set of bandwidth parameters to compare across the scaling experiments of Section 3 (Fig. S7). Again, poor choices of bandwidth values give misleading results, but bandwidth choices guided by the median heuristic generally perform well.
Finally, we vary the kernel choice across the scaling experiments of Section (Fig. using both a linear kernel (MMD,)) and the distance-induced kernel corresponding to the standard energy distance (MMDen). As expected, the linear kernel fails to distinguish distributions with matching means (2d-MoG) but performs reasonably well for distributions with mean-offscts even at high dimensions. The energy kernel performs similarly to the Gaussian kernel, without the added dependence sensitive hyperparameters.
38

---

Published in Transactions on Machine Learning Research (08/2024)
a
2-dim Mixture Gaussians
b
10-dim Gaussian
n-dim Gaussian
true
0.04
true shifted
true shifted
MMD0.5
approx.
MMD1
MMDS
0.02
0.02
0.00
0.00
0.00
true
true
true shifled
MMD1
0.04
approx.
0.05
shifted
MMD10
0.005
0.02
0.00
0.00
0.000
true
0.04
true
0.00050
MMD2
approx.
MMD6
shifted
MMD40
0.02
0.02
0.00025
true
shifted
0.00
0.00
0.00000
2000
4000
2000
4000
10'
10
sample size
sample size
dimensions
Figure S7: Evaluating the effect of varying bandwidth parameters Gaussian Kernel MMD for different sample sizes and dataset dimensionalities. (a,b) We compare sample sets with varying sample sizes (between 50 and 4k samples per set) 'true' distribution either with second dataset of the same distribution or with a sample standard deviation
from an approximated//hiee /shiiftte distribution. We show the mean and five runs of randomly sampled data. (a) MMD2 distance with varying bandwidth
parameters (0.5, 1, 2) for the 2d-MoG example shown in Fig. I compared to samples from unimodal Gaussian approximation with the same mean and covariance. (b) (b) MMD2 distance with varying bandwidth parameters (1, 3, 5) for a ten dimensional standard normal distribution, for which the first dimension is shifted by one for the shifted example. (c) MMD² distance with varying bandwidth parameters (5, 10, 40) based on 10k samples from a standard normal distributions with varying dimensions (between and 1000). As in (b) the first dimension shifted by one for the 'shifted' dataset. Here we show one run due to computational costs.
39

---

with a Gaussian kernel with Published in Transactions on Machine Learning Research (08/2024)
b
2-dim Mixture of Gaussians
10-dim Gaussian
n-dim Gaussian
Gaussian kernel
0.010
0.050
true approx.
true shifted
MMD1
MMD5
MMD10
0.005
0.025
true
shifted
0.000
0.000
2
true approx.
true
linear
MMDlin
MMDlin
shifted
MMDlin
0.2
0.5
true shifted
0.0
anergy-based
energy- -aaeededd
true approx.
0.4
true shifted
MMDen
MMDen
MMDen
shifted
0.2
0.0
0.0
0.0
2000
4000
2000
4000
10
10
sample
sample
dimensions
Figure S8: Comparison of Gaussian Kernel MMD to different MMD kernels without tunable parameters. We compare the performance of MMD as presented in Fig bandwidth parameters adjusted for each dataset (1,5,10) (top row) to an MMD implementation with linear kernel (middle row, MMD k(x, y) (x, ()) and an energy-distance based kernel, i.e., the kernel induced by the Euclidean distance Sejdinovic et al. (2013) (bottom row, MMD k(x,v) ||>]|| + ||>/|| ||. vll", with = 2). The experiments, parameters for the Gaussian kernel bandwidth (indicated in the y-labels), and sample sizes etc. are identical to Fig.
40

---

Published in Transactions on Machine Learning Research (08/2024)
A.13 Additional results for ImageNet generative models
We generated 50,000 images using unconditional dilfusion model explicitly trained on ImageNel. 64x64, well as 100,000 class-conditional images from a conditional variant of the same model (Dockhorn et al. 2022) (we refer to it as GENIE). For additional comparison, we also generated 50,000 images using a consistency model explicitly trained on ImageNet 64x64 (Song et al. 2023 (we refer to it as CM). We compare these generated images the Image.Ntt 64x64 test set.
For further comparison, also evaluated image-generative models not specifically trained to reproduce ImageNett but designed for general-purposs image generation, such Stable Diffusion and Midjourney RRobbach et al. 2022b Midjourney 2022). While these models might generate images that are more appealing to human observers, they do not necessarily produce images that align with the images contained in the ImageNet test set. We use the recently published million-scale dataset Genlmage (Zhu et al. 2023
Especially, we include the models BigGAN (Brock et al. 2019 ablated diffusion model (ADM: Dhariwal & Nichol (021]) Glide (Nichol ct al. 2022). Vector Quantized Diffusion Model (VQDM; Gu al. 2022).
Wukong (Wukong, 2022). Stable diffusion 1.5 (SD1.5; Rombach al. 2022b and Midjourney MMidjourney 2022
It's important to note that not all models are specifically trained to capture ImageNet 64x64 images. For instance, models like Stable Diffusion and Midjourney are trained on much larger datasets (Schuhmann et al,, 2022; Lin et al. 2014). Additionally, most of the models mentioned, except for ADM and BigGAN, are text-to-image generative models. To minimize significant distribution shifts, these models were prompted with the phrase 'photo of [ImageNet class]' Zhu et al. (2023). Furthermore, all the other models generate images of larger resolution, which we resized to 64x64. Hence, also only compare low-resolution features of natural images.
We evaluate each of the metrics on three random subsets, each consisting of 40,000 image embeddings. We show the average value for each model and metric in Table [54] Interestingly, both what considered "closest' to ImageNet, as well as the relative ranking differs for different metrics, although with some consistent trends. Overall, the most recent unconditional models trained on ImageNel 64x64 perform best, as expected. However, which one is considered best differs for different metrics. Metrics that consider similar features of the distribution ie., FID and MMD (statistics up to order 2 3) prefer GENIE, whereas universally consistent metrics do prefer CM (SWD and MMD&a). The estimated C2ST values differ based on the chosen classifier.
Overall, this analysis highlights that the choice metric (i.c., what features it compares) and the specific implementation details (such as the classifier in C2ST estimates) matter and can lead to varying results.
Table $4: Evaluating discrepancy to the ImageNet test Each presents various metrics computed the Inception v3 embeddings of images. Columns correspond to different generative models. The initial three models are trained on ImageNet 64x64, serving as the reference point for comparison. Subsequent models are trained on alternative datasets or higher-resolutinn versions. In bold, we highlight the lowest value for each section of the table.
GENIE
CM
BigGAN
ADM
Glide
VQDM
WK
SD1.5
Midjourney
FID
5.1 10
5.3 10"
1.2 10'
1.1
1.1 10'
9.8 10
1.1.10
1.2-10 10'
1.0-10'
SWD
2.3- 1
2.2 1
1
2.1
4.9-10-2
10-
4.8 1
5.0 1.02
4.5 10 2
MM 64
50
6.3
10
1.
1.810
10"
1.9
1.9- 10
1.7
MMD Ga
10
2.3 1
6.3-10
6.3
1071
62
1
12210-1
6.2-1071 10
6.4 10
6.1 1071
MMD) poly
1.1- 10
1.6- 10a
3.1-10' 10
10"
3.2-10
2.2
-.7
3.1- 10
2.9 10
C2ST
0.70
0.69
0.81
0.82
0.82
0.82
0.83
0.83
0.82
C2ST m
0.72
0.77
0.87
0.85
0.85
0.85
0.86
0.86
0.95
41

---

Published in Transactions on Machine Learning Research (08/2024)
A.14 Details about scientific application examples
For the motion discrimination lask, used the decision times of a single animal during both correct and erroneous trials with dot motion coherence of 12.8%, leading to one-dimensional dataset of 1023 samples. From these 80% were used as a train set and 20% as a tost set DDMs were implemented using the pyDDM toolbox (Shinn et al. 2020). DDM1 used a linear drift and exponential decision boundaries. In contrast, DDM2 used a constant drift and a constant decision boundary. Both were sampled 1,000 times create the two syuthetic datasets. The real chest X-ray dataset consists of 70,153 train samples and 25,596 test samples, the generated datasets from PGGAN and SD consist of 10,000 and 2,352 samples respectively.
In both applications we computed metrics between pairs of 10 random subsets from the compared distributions (scatter points on the violin plots). We computed the MMD with bandwidth of 50 for the medical imaging datasets and a bandwidth of 0.5 for the decision time dataset.
42

---

Published in Transactions on Machine Learning Research (08/2024)
References for Table S1
[4] J. H. Macke, L. Buesing, J. P. Cunningham, B. M. Yu, K. V. Shenoy, and M. Sahani. Empirical models of spiking in neural populations. In Advances in Neural Information Processing Systems, 2011.
[2] S. Linderman, M. Johnson, A. Miller, R. Adams, D. Blci, and L. Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017.
[3] C. Pandarinath, D. J. O'Shea, J. Collins, R. Jozefowicz, D. Stavisky, J. C. Kao, E. M. Trautmann, M. T. Kaufman, S. Ryu, L. R. Hochberg, M. Henderson, K. V. Shenoy, L. F. Abbott, and D. Sussillo. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature Methods, 2018.
[4] T. D. Kim, T. Z. Luo, T. Can, K. Krishnamurthy, J. W. Pillow, and C. D. Brody, Flow-field inference from neural data using deep recurrent networks. BioRriv, 2023.
5 M. Brenner, F. Hess, G. Koppe, and D. Durstewitz. Integrating multimodal data for joint generative modeling of complex dynamics. In Forly-forst International Conference on Machine Learning, 2024.
M. Molano-Mazon, A. Onken, E. Piasini, and S. Pauzeri. Syuthesizing realistic neural population activity patterns using generative adversarial networks. International Conference on Learning Representations, 2018.
(7) P. Ramesh, M. Atayi, and J. H. Macke. Adversarial training of neural encoding models on population spike trains. Real Neurons & Hidden Units: Future directions at the intersection of newroscience and artificial intelligence @ NewrIPS 2019, 2019.
[8] M. Bashiri, E. Walker, K.-K. Lurz A. Jagadish, T. Muhammad, Z. Ding, Z. Ding, A. Tolias, and F. Sinz. A flow-based latent state generative model of neural population responses to natural images. In Advances in Neural Information Processing Systems, 2021.
99 A. Schulz, J. Vetter, R. Gao, D. Morales, V. Lobato-Rios, P. Ramdya, P. J. Gonçalves, and J. H. Macke. Modcling conditional distributions of neural and behavioral data with masked variational autocncoders. bioRaiv, 2024.
[10] M. Pals, A. E. Sağtekin, F. Pei, M. Gloeckler, and J. H. Macke. Inferring stochastic low-rank recurrent neural networks from neural data. arXiv preprint orXiv:2406. 2024.
[11] J. Kapoor, A. Schulz, J. Vetter, F. Pei, R. Gao, and J. H. Macke. Latent diffusion for neural spiking data, arXiv preprint arXi:2407.08855, 2024.
[12] J. D. McCart, A. R. Sedler, C. Versteeg, D. Mifsud, M. based generation of neural activity from disentangled latent codes. arXiv preprint Xxw:2407.22199. 2024.
and C. Pandarinath. Diffusion-
[13] M. Dowling, Y. Zhao, and I. M. Park. Large-scale variational gaussian state-space models. arXiv preprint ar orxii=2408 01371, 2024.
[14] R. Lopez, J. Regier, M. B. Cole, M. I. Jordan, and N. Yosef. Deep generative modeling for single-cell transcriptomics. Nature methods, 2018
[15] M. Marouf, P. Machart, V. Bansal, C. Kilian, D. S. Magruder, C. F. Krebs, and S. Bonn. Realistic in silico generation and augmentation of single-cell ma-seq data using generative adversarial networks. Nature communiications, 2020.
[16] S. Lall, S. Ray, and S. Bandyopadhyay. LSH-GAN enables in-silico generation of cells for small sample high dimensional scRNA-seq data. Communications Biology, 2022.
[17] Y. Zinati, A. Takiddeen, and A. Emad. Groundgan: Grn-guided simulation of singlo-cell rna-seq data using causal generative adversarial networks. Nature Communications, 2024.
43

---

Published in Transactions on Machine Learning Research (08/2024)
[18] W. Tang, R. Liu, H. Wen, X. Dai, J. Ding, H. Li, W. Fan, Y. Xie, and J. Tang. A general single-cell analysis framework via conditional diffusion generative models. bioRwie, 2023.
[19] R. Cannoodt, W. Saclens, L. Deconinck, and Y. Sacys. Spearheading future omics analyses using dyngen, a multi-modal simulator of single cells. Nature Communications, 2021.
[20] O. Lindenbaum, Stanley, G. Wolf, and S. Krishnaswamy, Geometry based data generation. Advances Neural Information Processing Systems, 2018.
[21] L. Zappia, B. Phipson, and A. Oshlack. Splatter: simulation of single-cell RNA sequencing data. Genome biology, 2017.
[22] Н. L. Crowell, C. Soneson, P.-L. Germain, D. Calini, L. Collin, C. Raposo, D. Malhotra, and M. D. Robinson. Muscat detects subpopulation-specifee state transitions from multi-sample multi-condition single-cell transcriptomies data. Nalure communiicatinss, 2020.
[23] W. V. Li and J. J. Li. A statistical simulator sodesign for rational scRNA-seq experimental design. Bioinformalics, 2019.
[24]] T. Sun, D. Song, W. V. Li, and J. J. Li. scDesign2: a transparent simulator that generates high- high-fidelity single-cell gene expression count data with gene correlations captured. Genome biology, 2021.
[25] N. R. Stillman and R. Mayor. Generative models of morphogenesis in developmental biology. In Seminars in Cell & Developmental Biology, 2023.
[26] D. J. Waibel, E. Röell, B. Rieck, R. Giryes, and C. Marr. A diffusion model predicts 3d shapes from 2d microscopy images. In 2023 IEFE 20th International Symposium on Biomedical Imaging (ISBI), 2023.
[27] A. Zaritsky, A. R. Jamieson, E. S. Welf, A. Nevarez, J. Cillay, U. Eskiocak, B. L. Cantarel, and G. Danuser. Interpretable deep learning uncovers cellular properties in label-free live cell images that are predictive of highly metastatic melanoma. Cell systems, 2021.
[28] C. J. Soclistyo, G. Vallardi, G. Charras, and A. R. Lowe. Learning biophysical determinants of cell fate with deep neural networks. Nature Machine Intelligence, 2022.
[29] R. L. Satcher and C. F. Dowey. Thcoretical estimates of mechanical properties of the endothelial cell cytoskeleton. Biophysical journal, 1996.
[30] D. Stamenović, J. J. Fredberg, N. Wang, P. Butler, and D. E. Ingber. A microstructural approach to cytoskeletal mechanics based on tensegrity. Journal of Theoretical Biology, 1996.
[31] D. Stamenović and D. E. Ingber. Models of cytoskeletal mechanics of adherent cells. Biomechanics and modeling in mechanobiology, 2002.
[32] M. F. Coughlin and D. Stamenović. A prestressed cable network model of the adherent cell cytoskeleton. Biophysical journal, 2003.
[33] G. Jouvet, G. Cordonnier, B. Kim, M. Lüthi, A. Vieli, and A. Aschwanden. Deep learning speeds up ice flow modelling by several orders of magnitude. Journal of Glaciology, 2022.
[34] G. Jouvet.
Inversion of a stokes glacier flow model emulated by deep learning. Journal of Glaciology,
2023.
[35] G. Jouvet and G. Cordonnier. Ico-flow model emulator based on physics-informed deep learning. Journal of Glaciology, 2023.
[36] J. Bolibar, F. Sapienza, F. Maussion, R. Lguensat, B. Wouters, and F. Pérez. Universal differential equations for glacier ice flow modelling, Geoscienlifie Model Development, 2023.
[37] V. Verjans and A. Robel. Accelerating subglacial hydrology for ice sheet models with deep learning methods. Geophysical Research Letters, 2024.
44

---

Published in Transactions on Machine Learning Research (08/2024)
[38] R. Winkelmann, M. A. Martin, M. Hascloff, T. Albrecht, E. Bucler, C. Khroulev, and A. Levermann. The potsdam parallel ice sheet model (PISM-PIK) Part 1: Model description. The Cryosphere, 2011.
[39] E. Larour, H. Seroussi, M. Morlighem, and E. Rignot. Continental scale, high order, high spatial resolution, ice sheet modeling using the ice sheet system model (TSSM). Journal of Geophysical Reseorch: Earth Surface, 2012.
[40] O. Gagliardini, T. Zwinger, F. Gillet-Chaules G. Durand, L. Favier, B. de Fleurian, R. Greve, M. Malinen, C. Martin, P. Råback, J. Ruokolainen, M. Sacchettini, M. Schäfer, H. Seddik, and J. Thies. Capabilities and performance of elmer/ice, new-generation ice sheet model. Geoscientífic Model Development, 2013.
[41] R. Lam, A. Sanchez- Gonzalez, M. Willson, P. Wirnsberger, M. Fortunato, F. Alet, S. Ravuri, T. Ewalds, Z. Caton-Rosen, W. Hu, A. Merose, S. Hoyer, G. Holland, O. Vinyals, J. Stott, A. Pritzel, S. Mohamed, and P. Battaglia. Learning skillful medium-range global weather forecasting, Science, 2023.
[42] T. R. Shaham, T. Dekel, and T. Michacli. SinGAN: Learning generative model from a single natural image. In 2019 JEEE/CVE International Conference on Compuler Vision (ICCV), 2019.
[43] D. J. Gagne II, S. E. Haupt, W. D. Nychka, and G. Thompson. Interpretable doep learning for spatial analysis of severe hailstorms.. Monthly Weather Review, 2019.
[44] J. Côté, S. Gravel, A. Méthot, A. Patoine, M. Roch, and A. Staniforth. The operational CMC-MRB global environmental multiscale (GEM) model. Part i: Design considerations and formulation. Monthly Weather Review, 1998.
[45] T.. Zhou, S.-J. Lin, J.-H. Chen, L. M. Harris, X. Chen, and S. T. Rees. Toward convective-scale prediction within the next generation global prediction system. Bulletin of the American Meteorological Society, 2019.
[46] L. Magnusson, J.-R. Bidlot, M. Bonavita, A. R. Brown, P. A. Browne, G. D. Chiara, M. Dahoui, S. T. K. Lang, T. McNally, K. S. Mogensen, F. Pappenberger, F. Prates, F. Rabier, D. S. Richardson, F. Vitart, and S. Malardel. ECMWF activities for improved hurricane forecasts. Bulletin of the American Meteorological Society, 2019.
[47] D. M. Anstine and O. Isayev. Generative models as emerging paradigm in the chemical sciences. Journal of the American Chemical Society, 2023.
[48] R. Bombarlll,,,,lli, J. N. Wei, D. Duvenaud, J. M. Hernández-Lobato, B. Säuchez-Lengelingg D. Sheberla, J. Aguilera-Iparraguire,, T. D. Hirzel, R. P. Adams, and A. sspuru-Guzike Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 2018.
[49] R. Winter, F. Montanari, A. Steffen, H. Briem, F. Noé, and D.-A. Clevert. Efficient multi-objective molecular optimization in a continuous latent space. Chemical science, 2019.
[50] J. Lim, S. Ryu, J. W. Kim, and W. Y. Kim. Molecular generative model based on conditional variational autoencoder for de novo molecular design. Journal of cheminformatics, 2018.
[51] B. Sanchez-Lengeling C. Outeiral, G. L. Guimaraes, and A. Aspuru-Guzin Optimizing distributions over molecular space. an objective-reiforceed generative adversarial network for inverse-design chemistry (ORGANIC). ChemRaiv, 2017.
[22 N. De Cao and T. Kipf. MolGAN:
An implicit generative model for small molecular graphs. ICML 2018 workshop on Theoretical Foundations and Applicutions of Deep Generative Models, 2018.
[53] E. Hoogeboom, V. G. Satorras, C. Vignac, and M. Welling. Equivariant diffusion for molecule generation in 3d. In International conference machine learning, 2022.
[54] L. Huang, H. Zhang, T. Xu, and K.-C. Wong. Mdm: Molecular diffusion model for 3d molecule generation. In Proceedings of the AAAT Conference Artificial Intelligence, 2023.
45

---

Published in Transactions on Machine Learning Research (08/2024)
[55] L. Wu, C. Gong, X. Liu, M. Ye, and Q. Liu. Diffusion-baset molecule generation with informative prior bridges. Advances in Neural Information. Processing Systems, 2022.
[56] M. Popova, O. Isayev, and A. Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 2018.
[57] N. Ståhl, G. Falkman, A. Karlsson, G. Mathiason, and J. Bostrom. Deep reinforcement learning for multiparameter optimization novo drug design. Journal of chemical information and modeling, 2019.
[58] S. K. Gottipati, B. Sattarov, S. Niu, Y. Pathak, H. Wei, S. Liu, S. Blackburn, K. Thomas, C. Coley, J. Tang, el al. Learning to navigate the synthetically accessible chemical space using reinforcement learning. In International conference on machine learning, 2020.
[59] J. Regier, A. Miller, J. McAuliffe, R. Adams, M. Hoffman, D. Lang, D. Schlegel, and M. Prabhat. Celeste: Variational inference for generative model of astronomical images. In International Conference on Machine Learning, 2015.
[60] F. Lamusso, R. Mandelbaum, S. Ravanbakhsh, C.-L. Li, P. Freeman, and B. Póezos. Deep generative models for galaxy image simulations. Monthly Nolices of the Royal Astronomiial Sociely, 2021.
[61] M. J. Smith and J. E. Geach. Generative deep fields: arbitrarily sized, random synthetic astronomical images through deep learning. Monthly Notices of the Royal Astronomical Society, 2019.
[62] R. Mandelbaum, C. M. Hirala, A. Leauthaud, R. J. Massey, and J. Rhodes. Precision simulation of ground-based lensing data using observations from space. Monthly Notices of the Royal Astronomical Society, 2012.
[63] B. T. Rowe, M. Jarvis, R. Mandelbaum, G. M. Bernstein, J. Bosch, M. Simet, J. E. Meyers, T. Kacprzak, R. Nakajima, J. Zuntz, et al. GALSIM: the modular galaxy image simulation toolkit. Astronomy and Computing, 2015.
46


Reviews:
[{"review_id": "RFAqLIlV25", "reviewer": "Reviewer_oFeX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe draft is well-written, comprehensive and easy to follow.\nThe introduction and analysis of the metrics are systematic and provides practical guidance on how to use them.\nI believe it will be useful for new researchers to apply generative models to real-world, scientific problems in a more systematic way.\n\n## Weaknesses\nThe focus of the paper is sample-based distances but it was only made clear in the end of the 3rd paragraph in the introduction, i.e. neither the title or the introduction actually makes it clear that's the focus (and therefore these four distances are selected).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an comprehensive overview of sample-based distance metrics for evaluating generative models.\nIt covers four types most-commonly used metrics: sliced-Wasserstein distance (SW), classifier-based two-sample test (C2ST), maximum mean discrepancy (MMD) and Frechet Inception distance (FID).\nIt gives a very good overview (background + related work + intuition) to these metrics and discuss their pros and cons.\nIt conducts well-designed controllable experiments to study the statistical properties (i.e. how the metrics works when the number of samples and dimensionality change) as well as discusses the corresponding computational cost for practical use.\nFinally it studies two real use cases with guidance on how to interpret/use the metrics to analysis models in practice.\nI believe this manuscript is a good introduction to evaluation of generative models for people in the ML for science community.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main thing missing I think is the relationship between C2ST and density ratio estimation (DRE) and generative adversarial networks (GANs). I suggest the authors to include the following related work in section 2.2\n- DRE: the learned classifier has a density ratio interpretation of modeling $r(x) = p(x)/q(x)$ [1] and there are actually more methods than only using neural networks to learn such a classifier or to produce the density ratio estimates (e.g. kernel-mean embedding can be used to produce a method with closed-form solution [1,2])\n- Other f-divergences can be estimated based on density ratio estimators as all they needed are an estimated of $r(x)$ and they are used to learn various generative models effectively, e.g. Jensen–Shannon divergence in GANs [3,4], Kullback–Leibler divergence in [5,6], Pearson divergence in [7], etc.\n- Especially, the link of the classifier to the discriminator in GANs should be made clear.\n\n[1] Sugiyama, Masashi, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.\n\n[2] Huang, Jiayuan, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. \"Correcting sample selection bias by unlabeled data.\" Advances in neural information processing systems 19 (2006).\n\n[3] Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. \"Generative adversarial networks.\" Communications of the ACM 63, no. 11 (2020): 139-144.\n\n[4] Uehara, Masatoshi, Issei Sato, Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. \"Generative adversarial nets from a density ratio estimation perspective.\" arXiv preprint arXiv:1610.02920 (2016).\n\n[5] Titsias, Michalis K., and Francisco Ruiz. \"Unbiased implicit variational inference.\" In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 167-176. PMLR, 2019.\n\n[6] Huszár, Ferenc. \"Variational inference using implicit distributions.\" arXiv preprint arXiv:1702.08235 (2017).\n\n[7] Srivastava, Akash, Kai Xu, Michael U. Gutmann, and Charles Sutton. \"Generative ratio matching networks.\" arXiv preprint arXiv:1806.00101 (2018).", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t7hGdSHipJ", "reviewer": "Reviewer_DPTc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n- This paper is well-written and provides a good summary of existing measures in Section 2.\n\n## Weakness\n- The reviewer believes that the results provided by this study do not represent any new findings. This paper only summarized the existing methods and provided their performance for generative models. \n- Furthermore, the experiment in Fig. 9 for ImageNet shows the results with only one specific generative model, and Fig. 11 shows the results with only two models. It is unclear whether these results are consistent with the other generative models or specific to these models.\n- It is not discussed which distance is a good measure of the distance for the generative model. Since a \"good\" measure for generative models is one that is consistent with human perception, the \"good\" measure does not need to distinguish generated images from real images if humans recognize these generated images as real images. However, this work only discussed whether the various measures can distinguish the generated distributions and did not discuss which measure is appropriate.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work experimentally investigated the performance of four measures, Sliced Wasserstein distance, C2ST, MMD, and FID, for measuring the dissimilarity between real distribution and distribution generated by generative models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the above weakness section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SC03csjC9c", "reviewer": "Reviewer_Yh3B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n- The work overviews several popular metrics, including Freschet distance, classifier-based score and MMD-type metrics, as well as particular ways to use them (with/without the embedding space).\n- Experimental comparison of their scalability (w.r.t. dimensionality, number of samples, etc.) is present in several toy and real-world setups and some conclusions (recommendations to use this or that metric) are made based on it.\n\n**Weaknesses.**\nUnfortunately, I think that the presented study is not comprehensive enough and lacks a tremendous amount of important things that should be (in my opinion) included in such a survey. I will list such things below together with the weaknesses.\n\n- The paper pays practical attention to sample properties of the metrics (i.e., how many samples do you need to accurately estimate the metric), but barely mentions theoretical properties, namely, sample complexity. I think it is quite important to mention this aspect. What is the sample complexity? Which metrics have biased estimators and which have unbiased? Biased metrics a priory are expected to perform worse and be less reliable.\n- It would be good to have a single table summarizing the key theoretical and practical properties, advantages and limitations of the metrics studied. For example, the above-mentioned theoretical sample complexity, the theoretical computational complexity for N samples in dimension D, practical computational time, etc.\n- Studying the SW distance as a metric seems a little bit weird to me and rather far-fetched. Indeed, (as the authors say themself) “currently not extensively used in literature for evaluation”. Are there any existing papers which use it for evaluation (not for GAN training?). Also, I miss a clear discussion/evaluation related to the amount of required projections to accurately estimate SW.\n- The study of MMD is too quick in the sense that the authors just use a single kernel with a fixed (chosen according to some rule) bandwidth (+there is a small study in appendix of different width). I would expect a more thorough study of the effect of different kernels beyond the considered one. For example, there are kernels for which you do not need to tune the hyper-parameters such as the bandwidth, e.g., the energy-based kernels [1] defining the standard energy distance. How do such kernels perform?\n- One of the essential aspects of the generative modeling is to cover the entire distribution, i.e., not to drop/collapse any models of the distribution. In fact, this is the important part of the well-known generative modeling trilemma [2]. In the current paper, I did not find any convincing discussion of how the metrics take this aspect into account. (the word “collapse” appears only once in the paper). In fact, the community has invented several ways to address these aspects, e.g., the recall metric, but the current paper completely skips this and does not even mention.\n- The paper works with MMD and FID, but it seems like (probably I miss something) it does not directly mention the well-celebrated Kernel Inception distance, an MMD-based analog of FID (although the corresponding paper is cited).\n- For MMD, the authors give its clear formal definition in terms of distributions and then explain how to estimate it using the samples (provide an unbiased estimator). For the other metrics, the authors just directly write how to estimate them from samples or even explain this in words. This is not very good, especially in the case of Wasserstein, where the sample-based estimator is tremendously biased compared to the true Wasserstein between the underlying distributions. I think this might be misleading for the readers who are not familiar with such things and can lead to confusion. I recommend the authors the everywhere clearly explain the metric theoretically, and separately discuss the practical estimation aspects, just as they did in MMD.\n\n[1] Sejdinovic, D., Sriperumbudur, B., Gretton, A., & Fukumizu, K. (2013). Equivalence of distance-based and RKHS-based statistics in hypothesis testing. The annals of statistics, 2263-2291.\n\n[2] Xiao, Z., Kreis, K., & Vahdat, A. (2021, October). Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In International Conference on Learning Representations.\n\n**Other:**\n- What does this phrase mean? “However, a disadvantage of the Wasserstein distance is its transparency: The numerical value of the Wasserstein distance has no intuitive interpretation due to its definition in terms of optimal transport maps”\n- Do I understand correctly, that in all the cases (scientific applications) you compute the metric of generated images w.r.t. the test (hold-out set of data) which has been used for training? Can you also add the metric of test vs. train data in all the cases for comparison (Figs. 10., 11)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work provides an overview, some intuitive and experimental comparisons of measures of dissimilarity which are typically used to assess the quality of generative models in practice.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the authors should fix most of the issues to which I pointed above. Overall, I am quite confident that the current survey/guide is not comprehensive enough and misses a lot of crucial details and aspects about the evaluation metric for generative models.\n\nI would like to have a clear understanding how the current survey is better than any other existing surveys/related papers. What are the closely related surveys/guides? What is the novelty/improvement compared to them that will motivate the practitioner to read/use your survey rather than the existing ones? For example, a randomly googled recent related survey [3] already looks much more complete than the current paper (although it seems to be focused at slightly different aspects).\n\nLastly, the community also actively works on developing novel metrics/measures of dissimilarity to comapre probability distributions. It may be beneficial for the current survey/guide to at least mention them. For example, paper [3] proposes an alternatie to FID based on central kernel alignment (CKA). Paper [4] proposes some topology-based metrics, etc. More broadly, I think it would be beneficial for the current paper to at least mention such and related developments in the field to tell the practicioner some other directions to look at which may be particularly useful for his type of data and problem.\n\n[3] Yang, C., Zhang, Y., Bai, Q., Shen, Y., & Dai, B. (2023). Revisiting the evaluation of image synthesis with gans. Advances in Neural Information Processing Systems, 36, 9518-9542..\n\n[4] Barannikov, S., Trofimov, I., Sotnikov, G., Trimbach, E., Korotin, A., Filippov, A., & Burnaev, E. (2021). Manifold Topology Divergence: a Framework for Comparing Data Manifolds. Advances in neural information processing systems, 34, 7294-7305.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers emphasized that the paper is a survey paper that can serve as an entry point for readers not familiar with the subject. All three reviewers were inclined to accept the paper. One reviewer expressed concern that the new additions during the rebuttal phase somewhat decreased the natural flow of the paper. Acceptance with minor revisions is recommended:

1) Please double-check the spelling of the entire document. For example, see line 587: "comparisons.ue" should be "comparisons due."
2) Please consider improving the natural flow of the paper, which was somewhat disturbed after the rebuttal revision.
3) Please discuss more explicitly the standard limitations of the slicing-based method, when differences in two distributions are concentrated in a small number of features, making a randomly chosen direction in high-dimensional space almost always orthogonal to all of them. Currently, this is only briefly mentioned in Section 5, "Discussions," whereas Section 2.4, "Network-based measures," concerning another method, contains an explicit "Limitations" paragraph.
Decision: Accept (poster)