Title: GENEVAL: A BENCHMARK SUITE FOR EVALUATING GENERATIVE MODELS

Abstract: Generative models are important for several practical applications, from low level image processing tasks, to model-based planning in robotics. More generally, the study of generative models is motivated by the long-standing endeavor to model uncertainty and to discover structure by leveraging unlabeled data. Unfortunately, the lack of an ultimate task of interest has hindered progress in the field, as there is no established way to compare models and, often times, evaluation is based on mere visual inspection of samples drawn from such models. In this work, we aim at addressing this problem by introducing a new benchmark evaluation suite, dubbed GenEval. GenEval hosts a large array of distributions capturing many important properties of real datasets, yet in a controlled setting, such as lower intrinsic dimensionality, multi-modality, compositionality, independence and causal structure. Any model can be easily plugged for evaluation, provided it can generate samples. Our extensive evaluation suggests that different models have different strenghts, and that GenEval is a great tool to gain insights about how models and metrics work. We offer GenEval to the community 1 and believe that this benchmark will facilitate comparison and development of new generative models.

1. INTRODUCTION:

Modeling uncertainty is a fundamental problem for machine learning. In unsupervised settings, an ideal model should be able to describe all the possible events consistent with the provided context. In supervised settings, there is often not a single correct output for a given input, and models need to be able to express the space of correct outputs and their relative likelihood.

One standard approach to this problem is to build a probability model of the output space, or more generally, an energy-based model that assigns a score to every possible output. This kind of model is useful for comparing two possibilities, for example. Recently there have been a number of proposed models that do not necessarily assign any score to possible outputs, but using a pseudo-random number generator, produce hallucinations that in some way resemble the true outputs. This kind of model can be useful for examining possibilities.

In this work, we will consider the second kind of model, and take a "generative model" to be any construction that has a method for outputting data points using a pseudo-random number generator. In particular, we do not require "generative models" to be able to compute a probability of a data point. These kinds of models have become popular recently due to several works on Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) , that have demonstrated the generation of realistic images (Radford et al., 2015; Karras et al., 2017) .

However, this field has struggled to solve its zeroth problem (even as it has made progress on its first). Before the design of "better" models , it is necessary to agree upon good metrics and methodology to evaluate the quality of samples generated by a model. Part of the problem is that the downstream tasks of interest in the case of image generation have not been fully settled. Nevertheless, there have been a few works that make rigorous attempts to quantify the capabilities of these models, restricted to GANs parameterized as convolutional nets in the setting of natural images (Lucic et al., 2017; Huang et al., 2018) , and concurrently Kurach et al. (2018) .

In this work, we expand upon these with a suite of benchmark synthetic distributions that we use to evaluate not only GANs, but also other forms of generative models. We deliberately avoid convolutional networks on images with the aim of decoupling the benefits of various modeling paradigms from domain specific neural architectures. If we believe that generative models should be studied as generic tools for dealing with uncertainty (as opposed to methods of generating images), this decoupling is necessary. Additionally, practical, generic, and effective methods for measuring the distance between distributions in high dimensions with access only to samples do not currently exist. On the other hand, in some cases where one of the distributions has some special structure, we can effectively measure distortion, see an example of this in Appendix sec.A; using simple synthetic distributions allows us to be sure the distributions under consideration have the correct special structures. Analysis of which methods are able to model which kinds of special structures (e.g. independence, causality, multi-modality) is critical for understanding these methods; with "real" data not fully under our control, such analysis can be difficult.

A major contribution of this work is the GenEval toolbox to evaluate models. Thanks to its general API, new models can be easily plugged in and tested using a variety of metrics against popular baseline generative models. Likewise, it's very simple to add new distributions and evaluation metrics to the existing pool to highlight special properties of a model of interest.

2. RELATED WORK:

Much recent literature on generative modeling of continuous distributions has focused on modeling natural images. Restricting our attention to works aiming at generating samples, as opposed to scoring inputs, many recent models are based on Generative Adversarial Networks (Goodfellow et al., 2014; Arjovsky et al., 2017; Roth et al., 2017; Gulrajani et al., 2017; Denton et al., 2015; Karras et al., 2017) , which are capable of generating very realistic high resolution images, a task that had challenged the research community for several decades (Geman and Geman, 1984) .

For models trained on natural images, several metrics have been proposed, such as "inception score" (Salimans et al., 2016) and "Frechet inception" distance (Heusel et al., 2017) . Unfortunately, these metrics rely on yet another model for evaluation, and are known to only partially correlate with human judgment. To further complicate matters, generation scores not only depend on the metric but also on the choice of architecture of the generator, typically a rather sophisticated convolutional neural network. To summarize, comparison between different generative models is very difficult, and it is often based on visual inspection of a handful of model samples (Denton et al., 2015; Arjovsky et al., 2017) .

Several works have attempted to isolate the contribution of the estimation method by working in a controlled setting, e.g., by fitting the model to a mixture of Gaussians in two or three dimensions (Arjovsky et al., 2017; Roth et al., 2017; Gulrajani et al., 2017) , or in one dimension (Zaheer et al., 2017) . This paper can be seen as a more comprehensive, thorough and extensive study building upon these earlier attempts. While we share similar motivations, we do not restrict our focus to just GANs, but also include other popular models (Kingma and Welling, 2013; Dinh et al., 2016) .

Recently, Lucic et al. (2017) ; Kurach et al. (2018) have systematically compared different variants of GANs trained on natural images. Again, unlike this work, these focus only on GANs. Morover, we restrict our study to purely synthetic distributions with known properties which we leverage at evaluation time (or, from another perspective, we broaden our study to distributions beyond those coming from the image domain).

The choice of metrics for generative models has been intense topic of debate. Most works, overall those based on probabilistic modeling, report performance in terms of log-likelihood of samples generated by the data distribution according to the model. As discussed in earlier work (Theis et al., 2016) and in sec. 5, this is not a good metric in general, as it cannot be estimated by all models, e.g., GANs, and it is very susceptible to the scaling of the input. Interestingly, Lucic et al. (2017) propose a precision/recall metric to evaluate the quality of samples, where precision measures the distance of each model sample to the data manifold and recall measures the distance of each data point to the closest model generation. Unfortunately, such score disregards any weighting of the density, the fact that not all data points are equally likely and similarly, not all model samples have the same likelihood. Recognizing this issue, we borrow the "two-sample test" criterion introduced by Lopez-Paz and Oquab (2016) and extended by Huang et al. (2018) . This two-sample statistic is based on a similar motivation but it operates on actual samples, using implicit Monte Carlo weighting to take into account the density, and obviating the need for an explicit optimization over a data-manifold.

3. DISTRIBUTIONS:

The benchmark focuses on distributions with support on R d for some d. Their implementation in GenEval requires two methods: one for computing the log-density at a point, and one for sampling. GenEval implementation of distributions is modular, and allows combining distributions via products, mixtures, causal mappings, and isometric transforms. Next, we briefly describe the distributions we considered in this study.

Simple distributions Many of the distributions will be built from standard families of distributions, such as the multivariate Gaussian, and the Slab (or uniform) distribution with constant density on a rectanguloid in R d . Such distributions can be embedded into a lower dimensional space R q with q < d, yielding what we call a "flat" Guassian or Slab. For instance, the density function of a flat Slab is defined as:

p(x) ∝ χ R (O T (x -m)) + η(x),

where χ R is the uniform distribution with support in R = {x : 0 ≤ x i ≤ r i , i = 1, . . . , q}, O is a d × q matrix with orthogonal columns, and η is a Gaussian density with small variance. Mixtures Starting with K densities p 1 , ...., p k on R d , and a weighting vecor w on the simplex in R d , the density for a mixture is: p(x) = K i=1 w i p i (x). When the bulk of the mass of p i is disjoint from the bulk of the mass of p j for all i = j, the mixture model has a cluster structure. We show results with these kinds of distributions in fig. 1 , 2 and in Appendix sec. E. Distributions with causal structure Given a function f : R q → R d-q , q < d and a density p defined in R q , we can can get a density with causal structure on R d by considering

p(x) = p (x ind )η(x dep -f (x ind )),

where x ind are the independent first q coordinates of x (drawn from a certain distribution), and x dep is the last d -q dependent coordinates of x, and η is a noise distribution (for example zero-mean Gaussian with small variance). Note that these kinds of distributions have locally lower dimensional structure than d, especially when d is large compared to q. We show results with these kinds of distributions in fig. 5 . Locally low dimensional distributions We consider three types of locally low dimensional distributions: causal distributions where f above is parameterized by a random fully-connected ReLU net or a random quadratic polynomial (to simulate non-linear manifolds), and an "image-like" distribution of shifted bumps.

For the random neural-networks, we use one-hidden-layer ReLU networks with default PyTorch (Paszke et al., 2017) initializations. For the random quadratic polynomials, we take a sample for a two dimensional distribution and we project it into a d -2 dimensional space with a matrix of normally distributed coefficients, and then concatenate both vectors.

For the shifted bumps distribution, which is similar to the artificial triangle image distribution considered in (Lucic et al., 2017) , we take a side-length s, a set of radii r 0 , ..., r l , and consider all shifts of an r i × r i square on a s × s background. These form a set of s 2 * l points in R s 2

; we take a random orthogonal projection of this set to R d and place a ball of mass around each projected point. Experiments with these distributions are shown in fig. 3 . Distributions with independence structures We can increase the intrinsic dimension of any of these previous distributions by combining them as a product: p(x) = Π K j=1 p i (x), where p 1 , ..., p K are the basic components which have non-zero support on a disojint subset of coordinates. Note that models are not provided with such information at training time. We report results with product distributions in fig. 4 and in Appendix.

4. GENERATIVE MODELS:

In this section we first describe some classical baselines, meant to provide upper and lower-bound performances on each distribution, and then some neural-network based models.

Baselines::

The oracle baseline is the data distribution itself. Given a metric assessing the discrepancy between samples drawn from the data and the model distributions (see sec. 5), oracle samples provide a lower bound on the error of an ideal model perfectly fitting the data. Ideal metrics would report an error close to 0 for oracle samples.

The next baseline we consider is the kernel density estimator (KDE) (Rosenblatt, 1956; Parzen, 1962) which places a Gaussian bump of a certain width around each training point. KDE essentially memorizes the training set, and therefore, it is expected to provide great fitting to the training data but poor generalization, overall in high dimensional spaces where the curse of dimensionality would require a vast amount of training samples to finely cover the data distribution.

Next, we have the multivariate Gaussian distribution fitted by maximum likelihood (Pedregosa et al., 2011) , meant to provide some sort of upper bound on the error, and the mixture of Gaussians with k components (MoG k ), which we fit by using the expectation-maximization algorithm (Dempster et al., 1977) . We fix the number of Gaussians in the mixture to 10 in all experiments.

Neural Models: Among the neural models, we consider the Variational Auto-Encoder (VAE) (Kingma and Welling, 2013) , the real non-volume preserving density estimator (RNVP) (Dinh et al., 2016) and Generative Adversarial Network (GAN) (Goodfellow et al., 2014) with some of its variants, namely Wasserstein GAN (WGAN) (Arjovsky et al., 2017) , WGAN with Gradient Penalty (WGAN-GP) (Gulrajani et al., 2017) and GAN with Noise Regularization (GAN-NR) (Roth et al., 2017) . See Appendixfor a brief review of these methods.

5. DISTORTION STATISTICS:

Since all models must be capable of drawing samples, we measure fitting error in terms of the distortion between two sets of points: samples drawn from the data distribution and from the model distribution. We will discuss several measures of fidelity to a distribution; and none of them are generically useful. Measuring the distortion between real-valued distributions in high (or even not-so-high) dimensions continues to be an unsolved problem, especially if we add constraints on computational efficiency. This is one of the reasons to use artificial data with special properties we control: although none of the metrics are generally useful, they can all be useful when the distributions have appropriate structure (see example in Appendix sec. A).

Optimal Transport:

The Optimal Transport distance (OT) u ρ between sets S and T of points in a metric space with metric ρ, is defined by:

u ρ (S, T ) = min i,j λ ij ρ(s i , t j ) s.t. i λ ij = 1 and j λ ij = 1 and λ ij ≥ 0 (1)

We use the POT package2 for computing the optimal transport, and assume ρ is Euclidean l 2 for the rest of this work, and drop it from the notation.

Sampling a fixed number of points from two distributions P and Q and computing the OT distance between the samples does not give a distance on distributions (even if it were well defined, it would not even give "distance" 0 from a distribution to itself), but it does correspond to an estimator of the OT distance between P and Q. However, in high dimensions, with few samples, this estimator can have possibly counter-intuitive behaviors. For example (Arora et al., 2017) , suppose we sample N points S and N points S from the uniform distribution over the unit sphere in R d , where d is large enough to see concentration-of-measure effects, and then N points T from δ 0 (the point mass at the origin). If N is small compared to d, then the points in S and S are almost orthogonal, and so have distance roughly √ 2 between them. On the other hand, the distance of any point in S to a point sampled from T is 1. It is thus likely that u(S, S ) > u(S, T ) even though S is sampled from the same distribution as S and T is not.

On the other hand, OT has some pleasing properties. It is continuous with respect to perturbations of the points in the sets. Furthermore, if two sets of points covers the same low-dimensional manifold in a high dimensional space, the OT distance between them will be relatively small.

In the tables below, when using OT to measure the success of a generative model, we will always report the "oracle" OT score given by comparing two sets of samples from the true distribution. When the oracle scores are relatively high, the relative results should be taken cautiously.

Nearest Neighbor Two-Sample Statistic Two set of points, S ∼ p d and T ∼ p g , are close if the chance that a point from S has nearest neighbor belonging to T is 50% and vice versa. This gives the intuition behind the nearest neighbor two-sample statistic (2S) (Huang et al., 2018) . In this work, we use a version similar to the one in Huang et al. (2018) .

Assume |S| = |T | = N . For each point x in S or T , define n(x) to be 3 n(x) = 1, if min y∈S ||x -y|| < min y∈T ||x -y|| 0, otherwise. .

Then, we define the distance as the sum of the deviations from the optimal rate:

v(S, T ) = 1/2 - x∈S n(x)/N + 1/2 - x∈T (1 -n(x))/N (2)

As N gets larger, it becomes easier to distinguish the distributions via this statistic; but different values of N may lead to different quality orderings. As the dimension of the distribution gets higher (for fixed N ), it becomes harder to distinguish distributions via this statistic. As with OT, we will always show the oracle value for a distribution when showing results of generative models.

Log-Likelihood In this work log-likelihood (LL) is estimated on the samples drawn from the model using the (known) data distribution -since we do not require models to necessarily be able to estimate data log-likelihood. This gives a notion of how likely points generated by the model are, regardless of the overall fit. In other words, samples drawn from a model assigning all its mass to the mode of the data distribution will have even higher likelihood that samples drawn from the actual data distribution.

5.1. SPECIAL STATISTICS:

Our distributions have special structure that we can take advantage of, to better measure success in modeling that distribution.

Mode coverage (MC) Assuming a mixture distribution for the data and a uniform distribution over modes, we measure whether samples generated from the model have even coverage of the clusters. We report "mode coverage" (MC) as the perpelxity of the mode assignments: a) , where H is the entropy of the cluster assignment distribution a.

M C = 2 H(

Causal Discovery Assuming a causal distribution (see sec. 3) and the ability of the model to perform conditional inference of a set of variables given the complementary set, we measure how close (in l 2 ) the recovered dependent coordinates are from what the ground truth values given a set of independent coordinates drawn from the true underlying distribution.

For all the neural models with a latent space, we estimate the missing variables by optimizing over the latent variable with an l 2 reconstruction loss over the observed coordinates, starting from a random point. For KDE, we find the nearest point from the training set in the constrained coordinates, and return the remaining coordinates of that training point. For a mixture of Guassians, we take the nearest mean in the constrained coordinates among all components, and then return the maximum-likelihood estimate from that Gaussian of the remaining coordinates. See fig. 5 for results using this metric. Independence Test Measuring the independence of high-dimensional real valued distributions is challenging. In order to get some idea of the ability of the various models to detect independence, we will restrict our attention to distributions that are products of distributions that are easy to vectorquantize, and use categorical tests for independence on the quantized values. That is, suppose we have a distribution p = K i=1 p i where p i is supported on the set of coordinates c i . Further suppose we have clustered p i into clusters C ij for j ∈ {1, ..., n i }, and that we have L samples X from a model. Then, for each coordinate group c i , we make a table of size n i × (n 1 n 2 ...n i-1 n i+1 ...n K ) with the counts N s,t of the number of points in X that landed in each cell. Here, s indexes rows of this table, corresponding to the n i centroids C i1 , ..., C ini for the points projected down to coordinate group c i , and t indexes columns of the table, corresponding to the product of all other clusters in all other coordinate groups.Denote by M t = s N s,t /L and M s = t N s,t /L; we expect that N s,t /L ∼ M s M t , so we take:

χ 2 = s,t (N s,t /L -M s M t ) 2 .

Independent OT and 2S There is a partial remedy for dealing with the weaknesses of the distortion measures in high dimensions for distributions that we know have independence structure, see sec. 3. Suppose that the distribution p we are interested in has support in R d . Further suppose there are groups of coordinates I 0 , I 1 , ...I L partitioning [1, ..., d] such that p Ij ⊥ p Ii for i = j. Here p I is the distribution given by projecting p onto the subspace spanned by the coordinates in I. Then, for some distortion measure u, we can use the set distance given by:

EQUATION

where S Ii is the projection of S onto I i (and likewise for T ).

In building such a distortion measure, we are using the knowledge of the true distribution. However, because we are building the distributions in the benchmark, we are free to use this information in measuring distortion, even if the models should not get access to it during training. Furthermore, note that this kind of distortion measure only can tell the difference between distributions that disagree on the marginals of the groups. Even if it shows no distortion between two distributions, they may still be different. On the other hand, because each group has smaller dimension, the distortion measure is less pre-disposed to show everything being different from everything else.

6. GENEVAL:

GenEval is written in Python and it consists of three main components: models, distributions, and metrics.

Models must define fit(X) and sample(N) methods, the former to train the model on a dataset of points X, the latter to draw N samples. Models can optionally define a method to conditionally sample data given values for a set of fixed coordinates. Thanks to the general interface and modularity of GenEval, we simply incorporated the original implementations by the authors of the models whenever available. As a result, GenEval does not enforce any specific machine learning framework.

For VAE(vae), GAN(gan, b), WGAN(wga, b), WGAN-GP(wga, a) and GAN-NR(gan, a) we used PyTorch (Paszke et al., 2017) , while for RNVP(nvp) we used TensorFlow (Abadi et al., 2015) . For Mixture of Gaussians and Kernel Density Estimation we employ the scikit-learn (Pedregosa et al., 2011) package.

Each distribution class must define the methods: sample(N) and logprob(X). Mixture distributions may also optionally define a method to estimate the most likely cluster assignment to each input data point, and product distributions can specify their components. New distributions can easily be defined via composition directly in configuration files specifying which distribution to test on. For instance, the library contains classes for a slab, a mixture of arbitrary distributions, and affine transform. Given these, one may define on-the-fly a distribution over the surface of a rotated 3D box in some higher dimensional space.

Metrics methods take as input two sets of samples (from the data distribution and from the model) and output a scalar value. Some metrics may optionally take as input a trained model and the data distribution to compute distribution dependent metrics like mode coverage.

Models and distributions are passed as input to GenEval via two Python script configuration files, which specify all their eventual hyper-parameters. Model hyper-parameters can also be specified via lists, which are used by GenEval to run a grid search over hyper-parameter values. A user may then run GenEval with configuration files that specify several distributions and several models. GenEval then launches training and evaluation for the cartesian product of all possible combinations of model and distribution and report results on a table, as those shown in the Appendix.

7. EXPERIMENTS:

We used GenEval to compare the models described in sec. 4 on several distributions. For each distribution, GenEval first runs a grid search over hyper-parameters, see for instance the configuration files in Appendix sec. D. To produce the results in this section for every method we ran a very extensive grid search over hyper-parameters, which reached about 20, 000 configurations for some GAN variants; and was in the thousands for all neural methods. Afterwards, GenEval compares models across all metrics and compiles tables, such as those in Appendix E.

Unless otherwise stated, all experiments we discuss next have used 10,000 training samples, and unless otherwise specified in the figure, 1000 validation and test samples. On the figures showing results with OT, OT was used as a cross-validation metric for the hyperparameter search; and 2S was used for figures showing results with 2S. For all other figures, cross validation was done using 2S.

Fig 1 shows two-sample results when the true distribution is a mixture of Gaussians with varying number of components in the mixture (left), variance (bottom) and intrinsic dimensionality (top versus bottom). First, we observe that model performance is bounded by the oracle, as expected, with a nearly 0 distortion. A mixture of Gaussian with 10 components achieves the lowest distortion when the number of components in the true distribution is less than 10, but distortion degrades Figure 4 : 2S and OT on products of five identical distributions, each of which has support near a two or three dimensional embedded manifold in R 10 . In this figure, we use uind from eq. 3 (using the true groups of independent coordinates) to measure distortion and to validate models.

very rapidly as soon as the true number of components increases. The KDE baseline instead has distortion which is constant with respect to number of components and variance4 . KDE often leads to competitive distortion compared to neural models, yet much higher than the oracle. Second, in this experiment, GAN variants do not perform reliably better than the original GAN. Third, we observe that none of the neural models perform well, despite the relative simplicity of the distribution.

Modeling distributions with more components, tighter variance and lower intrinsic dimensionality makes the learning problem considerably harder for these models. The results are somewhat different if we cross-validate and evaluate according to OT, as shown in fig. 2 . With this statistic, there is no performance gap between the oracle and KDE, highlighting the importance of the choice of metric when comparing methods. Oracle performance according to OT degrades as a function of the number of components (left half of Figure 2 ), and especially as the intrinsic variance increases (bottom right of Figure 2 ), but not when the intrinsic dimensionality is lower (top right of Figure 2 ). Thus, as suggested by the example in 5, sampled OT can be an unreliable measure of success when the true distribution fills out space in high dimensions.

Next, we evaluate on various manifold distribution and product of manifold distributions, see fig. 3 and 4 . One interesting observation is that OT and 2S do not correlate very well (see discussion in sec. 5). On these tasks, Real NVP performs well in terms of the nearest neighbor two-sample statistic, followed by MoG 10 , with other methods performing worse. In terms of OT, KDE does best on the products of manifolds, and VAE does the best on the manifolds. In fact, VAE does better than oracle there (see two bottom left plots in fig. 3 and the note in 5), suggesting it is denoising the true distribution.

In the next experiment reported in fig. 5 , we analyze the ability of the models to discover causal structure on manifold distributions. We do not report the results of Real NVP here because we cannot make a conditional estimate in the same way as the other neural models. We observe that all models have hard time beating the MoG 10 baseline in this case.

Figure 6 : We consider product of 5 (top) and 8 (bottom) distributions, each of which is a mixture of three

Gaussians in 12 dimensions. The χ 2 metric, see sec. 5.1, measures the extent by which models have captured the independence structure of the product distribution. Note that higher is better for mode coverage, unlike the other metrics. 2S is computed using 1K samples, while MC and χ 2 use 50K samples

Finally, we look at a product of independent Gaussian mixture distributions in fig. 6 . None of the models are able to succesfully fit this distribution, although WGAN-GP and Real NVP do the best. KDE in particular fails to fit this distribution, both in terms of two-sample and in terms of the χ 2 independence test. We can also see that all of the GAN variants improve on the mode coverage of the vanilla GAN.

8. DISCUSSION:

Perhaps to the delight of neural generative model skeptics, one sees that on almost all distributions and all metrics that we consider, one of KDE or mixture of 10 Gaussians are competitive with (and often superior to) the neural models. These take a tiny fraction of the training cost, have essentially no hyper-parameter sweep, and have simple, well understood fitting routines. However, the neural models, especially GAN, are designed to be used where l 2 in the observation space makes little sense as a metric. On most of our examples l 2 in the observation space is locally meaningful, and most of our metrics rely on l 2 to be (at least) locally meaningful. Thus we consider success at our benchmarks neither necessary nor sufficient for a model to be good at the perhaps more complex tasks for which neural models are designed. We discourage neural model skeptics from dismissing neural models based on these results: worse results on simple tasks against unscaleable baselines better adapted to the simple tasks does not necessarily mean an approach should be abandoned.

On the other hand, neural generative model boosters should take heed of these results. They show that on datasets that are not images, with networks that are not convolutional networks, neural models do not do well. Moreover, specific to GANs, our results corroborate the results of Zaheer et al. (2017) ; Arora and Zhang (2017) , and suggest that GANs do not necessarily learn to sample from the target distribution. Furthermore, some protocols purported to improve GANs in the image setting do not reliably improve results across the metrics and datasets shown here (although cluster coverage does seem to be reliably improved). This is natural, as practitioners have spent much effort tuning models for performance in the image domain, and of course neural models will do poorly without well adapted architectures5 . Nevertheless, most works trying to improve or understand GANs (empirically or theoretically) discuss the training protocol independently of the relationships between the inductive biases of the neural architectures and the properties of the distributions to be modeled. Our results here suggest that these relationships cannot be ignored when studying GANs.

We also see that while no single neural model is all-around superior to the others, Real NVP does do better than the other models in many cases, especially in terms of the two-sample statistic. However, it does not dominate, for example doing worse at products of manifolds in terms of OT, see fig. 4 , and worse on mixtures of Gaussians with large numbers of components, see fig. 1 6 .

We can also see other quirks of the models. For example, we can see that "denoising" effects of a VAE can make it appear to be a good model on some datasets and metrics, but poor in others. It is one of the better performing neural models in fig. 5 when measuring causal error, showing it has learned the manifold and the functional dependence. But at the same time, it is bad in terms of twosample (fig. 3 ); perhaps because it is collapsing the two free components in the output. More generally, these kinds of effects show that in general, it is important to consider multiple metrics. The gold standard should be a success in a downstream task of interest, but in the absence of such a task, looking at a single metric to judge success can be misleading.

A BLOCK INDEPENDENCE AND DISTORTION MEASURE RESOLUTION:

We give a simple example showing how eq. 3 can increase the accuracy, or more precisely, the resolution of a distortion measure. We take a product of 10 mixtures of Gaussians in R 12 , each with 3 components, with modes randomly distributed over the unit sphere in R 12 , as in fig. 6 . The overall ambient dimension is 120.

Measuring OT against a single best fit Gaussian, which is clearly a poor model of the original distribution, without leveraging the groups structure (see again eq. 3), gives a distortion of 7.3 for the oracle samples versus 8.6 for the samples from the Gaussian. If we do leverage information about the independence structure among the groups, we get a distortion of 5.9 for the oracle versus 19.3 for the Gaussian. Raising the number of mixtures in the product to 30 gives 17.4 vs 17.6 without groups and 17.5 vs 54.5 with.

It is therefore essential even in relatively low dimensions to leverage the special structure of the distributions in order to assess the real quality of the sameples generated by a model.

B OTHER METRICS:

In addition to the metrics introduced in sec. 5,we also considered the Hausdorff distance.

Hausdorff Distance The Hausdorff distance (H) h ρ ∞ between sets S and T of points in a metric space with metric ρ, is defined by:

EQUATION

As in the case of optimal transport, this defines a metric between sets; but when we sample from the distributions and compare the distance between the samples, we do not get a metric on the distributions. We will denote by H ∞ (P, Q; N ) the statistic obtained by sampling N points from P , N points from Q, and computing the Hasudorff distance between the sampled points. We will show results with an average version of this statistic: by H 2 (P, Q; N ) we take the average minimal square distance, instead of the max. That is, define:

EQUATION

and set the statistic H 2 (P, Q; N ) to be h 2 (S, T ) for N points S sampled from P and N points T sampled from Q. Note that h 2 is not even a metric on sets, as it does not satisfy the triangle inequality.

The Hausdorff statistics should be taken as a measure of the difference of support between distributions. That is, as the number of sampled points gets large, two distribution with the same support (but perhaps very different densities on that support) will have relatively small difference in the Hausdorff statistic.

C ANALYSES OF TWO-SAMPLE TEST:

The Two-Sample Test depends on the number of points sampled from the distributions. We analyze how number of test points and number of training points affects the metric. We use random quadratic polynomial distribution and a distributions of shifted bumps as shown in fig. 7 and as described in sec. 3. For both distributions we observe a similar pattern. As the number of points goes up the errors also goes up for all the models. However, the degradation is much more prominent for KDE.

The same applied to the size of the training set: neural models may generalize from less amount of training data. 

D HYPER-PARAMETER SEARCH:

As mentioned in the section 6, model configurations for each experiment are defined in configuration files. Each file defines a MODELS variable with a list of dictionaries describing the model. Each dictionary has three keys: name, model (refers to a model class), and args. A user may want to re-use the same model in several cases to evaluate impact of different subsets of parameters. E.g., all flavors of the GAN models implemented in a single class and evaluated separately. Hyper parameters for each case are defined in args dictionary that maps parameter to one or more possible values.

All our models use multilayer perceptron as the base architecture. The default activation function is ReLU, but in order to help the model to sample from a mixture of gaussians, we also considered q-ary multi maxout activation that is a max pooling over groups of q units. The activation function is determined by maxout parameter. Zero stands for ReLU, while positive values stand of q-ary multi maxout. Parameter --nz defines the size of the latent space. By default GAN models use RMSProp optimizer unless --adam is specified. RealNVP and VAE use Adam optimizer by default.

We followed the choice in the original implementations here. (Arjovsky et al., 2017) noticed that for better results in training WGAN several discriminator updates should be done per generator updates. Besides, discriminator should get 100 extra updates for first 25 steps and every 500th step.

We captures these heuristics in --Diters and --boost discriminator flags.

For RealNVP we use only channel based masking with 8 coupling layers (defined by --chain length). Each layer is implemented by a multilayer perceptron with optional batch and weight normalization.

Below we show the complete configuration file used for multimodal distributions. 

:

Available at: coming soon.

http://pot.readthedocs.io/en/stable/

In our implementation, if the quantities miny∈S||x -y|| = miny∈T ||x -y|| we choose n(x) randomly in {0, 1}.

This is essentially a function of the number of points used for computing the two-sample statistic vs. the number of training points. The nearest neighbors of real points may be generated or real; KDE is essentially perfect on this side of the distortion statistic. But the neighbors of generated points will almost certainly be generated as the number of test points increases, as the neighbor will have been generated from the Gaussian about the same training point.

Note however we have checked several of the distributions where neural models perform poorly, and explicitly constructed by hand neural generators that well approximate these distributions. Thus the problem is not merely the generator being out of the search space.

All of the models are doing poorly here, though.

Figures and Tables:

Figure 1: Two Sample test on Mixture Gaussians. Top: each Gaussian component has intrinsic dimensionality equal to 5, with a spherical covariance (in 5d) rotated to a random orientation 50d and non-zero mean placed at random; the ambient dimension is 50. Bottom: each Gaussian component is defined in a 50 dimensional space, and it has spherical covariance and a non-zero mean placed at random in that space. Left: varying the number of components in the mixture from 2 to 50. Right: varying the variance of each component; note that for the flat mixture (Top) only the variance in the intrinsic dimension changes.

Figure 2: Same as above but using OT for both cross-validation and evaluation.

Figure 3: Two-sample statistics and optimal transport on two and three dimensional embedded manifolds in R 50 .

Figure 5: Prediction error of dependent variables in causal distribution, see sec. 3 and 5.1. Hyperparameters are chosen by best OT.

Figure 7: Two-Sample test for different number of test points for a random quadratic polynomial distribution (top) and for a distribution of shifted bumps (bottom), when training with 1k (left) and 10k (right) samples. Lower is better.

Mixture distributions with 10000 training points using OT metric.

Table content: <table><tr><td/><td>oracle</td><td>KDE</td><td>Gaussian</td><td>MOG</td><td>VAE</td><td>RealNVP</td><td>GAN</td><td>WGAN</td><td>WGAN-GP</td><td>GAN-NR</td></tr><tr><td>MoG2 ad50 id5 mtC r0.03</td><td>0.089</td><td>0.114</td><td>0.536</td><td>0.059</td><td>0.316</td><td>0.196</td><td>0.139</td><td>0.316</td><td>0.154</td><td>0.111</td></tr><tr><td>MoG5 ad50 id5 mtC r0.03</td><td>0.104</td><td>0.089</td><td>0.531</td><td>0.108</td><td>0.770</td><td>0.197</td><td>0.407</td><td>0.484</td><td>0.154</td><td>0.142</td></tr><tr><td>MoG10 ad50 id5 mtC r0.03</td><td>0.106</td><td>0.115</td><td>0.453</td><td>0.112</td><td>0.654</td><td>0.233</td><td>0.160</td><td>0.314</td><td>0.144</td><td>0.158</td></tr><tr><td>MoG20 ad50 id5 mtC r0.03</td><td>0.127</td><td>0.144</td><td>0.426</td><td>0.151</td><td>0.674</td><td>0.276</td><td>0.170</td><td>0.918</td><td>0.165</td><td>0.162</td></tr><tr><td>MoG50 ad50 id5 mtC r0.03</td><td>0.152</td><td>0.150</td><td>0.421</td><td>0.160</td><td>0.648</td><td>0.298</td><td>0.151</td><td>0.456</td><td>0.180</td><td>0.222</td></tr><tr><td>MoG10 ad50 id5 mtC r0.1</td><td>0.160</td><td>0.168</td><td>0.499</td><td>0.176</td><td>0.678</td><td>0.323</td><td>0.243</td><td>0.469</td><td>0.186</td><td>0.275</td></tr><tr><td>MoG10 ad50 id5 mtC r0.3</td><td>0.250</td><td>0.258</td><td>0.611</td><td>0.280</td><td>0.766</td><td>0.404</td><td>0.391</td><td>0.581</td><td>0.289</td><td>0.306</td></tr><tr><td>MoG10 ad50 id5 mtC r1.0</td><td>0.429</td><td>0.440</td><td>0.885</td><td>0.436</td><td>0.962</td><td>0.553</td><td>0.487</td><td>0.815</td><td>0.491</td><td>0.492</td></tr><tr><td>MoG10 ad50 id5 mtC r3.0</td><td>0.721</td><td>0.741</td><td>1.369</td><td>0.802</td><td>1.326</td><td>0.852</td><td>0.837</td><td>1.136</td><td>0.906</td><td>0.782</td></tr><tr><td>MoG2 ad50 id5 mtR r0.03</td><td>0.257</td><td>0.392</td><td>2.774</td><td>0.103</td><td>0.477</td><td>0.239</td><td>5.556</td><td>8.248</td><td>0.336</td><td>5.424</td></tr><tr><td>MoG5 ad50 id5 mtR r0.03</td><td>0.435</td><td>0.275</td><td>5.172</td><td>0.252</td><td>1.204</td><td>1.084</td><td>1.346</td><td>4.635</td><td>0.638</td><td>1.340</td></tr><tr><td>MoG10 ad50 id5 mtR r0.03</td><td>0.338</td><td>0.632</td><td>6.353</td><td>0.313</td><td>1.816</td><td>1.651</td><td>2.887</td><td>2.736</td><td>1.000</td><td>4.268</td></tr><tr><td>MoG20 ad50 id5 mtR r0.03</td><td>0.886</td><td>0.911</td><td>7.047</td><td>2.919</td><td>2.607</td><td>2.128</td><td>4.869</td><td>3.239</td><td>1.890</td><td>3.787</td></tr><tr><td>MoG50 ad50 id5 mtR r0.03</td><td>1.328</td><td>1.202</td><td>7.523</td><td>5.254</td><td>3.199</td><td>4.149</td><td>6.429</td><td>5.750</td><td>2.256</td><td>5.941</td></tr><tr><td>MoG10 ad50 id5 mtR r0.1</td><td>0.391</td><td>0.682</td><td>6.336</td><td>0.367</td><td>1.935</td><td>1.572</td><td>2.862</td><td>4.341</td><td>0.887</td><td>3.928</td></tr><tr><td>MoG10 ad50 id5 mtR r0.3</td><td>0.478</td><td>0.766</td><td>6.283</td><td>0.622</td><td>1.662</td><td>1.640</td><td>2.891</td><td>3.175</td><td>1.368</td><td>2.764</td></tr><tr><td>MoG10 ad50 id5 mtR r1.0</td><td>0.648</td><td>0.930</td><td>6.264</td><td>0.835</td><td>2.095</td><td>2.194</td><td>2.527</td><td>3.534</td><td>1.148</td><td>2.826</td></tr><tr><td>MoG10 ad50 id5 mtR r3.0</td><td>0.926</td><td>1.201</td><td>6.291</td><td>1.094</td><td>2.404</td><td>2.421</td><td>3.401</td><td>2.989</td><td>1.796</td><td>3.536</td></tr><tr><td>MoG2 ad50 id50 mtC r0.03</td><td>0.452</td><td>0.473</td><td>0.689</td><td>0.426</td><td>0.488</td><td>0.521</td><td>1.009</td><td>1.111</td><td>1.054</td><td>0.495</td></tr><tr><td>MoG5 ad50 id50 mtC r0.03</td><td>0.464</td><td>0.455</td><td>0.668</td><td>0.477</td><td>0.772</td><td>0.517</td><td>0.883</td><td>0.639</td><td>0.428</td><td>0.443</td></tr><tr><td>MoG10 ad50 id50 mtC r0.03</td><td>0.464</td><td>0.467</td><td>0.624</td><td>0.463</td><td>0.797</td><td>0.536</td><td>0.827</td><td>0.678</td><td>0.613</td><td>0.425</td></tr><tr><td>MoG20 ad50 id50 mtC r0.03</td><td>0.469</td><td>0.476</td><td>0.613</td><td>0.465</td><td>0.798</td><td>0.520</td><td>0.450</td><td>0.573</td><td>0.462</td><td>0.454</td></tr><tr><td>MoG50 ad50 id50 mtC r0.03</td><td>0.471</td><td>0.474</td><td>0.613</td><td>0.465</td><td>0.825</td><td>0.503</td><td>0.442</td><td>0.486</td><td>0.404</td><td>0.436</td></tr><tr><td>MoG10 ad50 id50 mtC r0.1</td><td>0.824</td><td>0.826</td><td>0.919</td><td>0.825</td><td>0.989</td><td>0.853</td><td>0.750</td><td>0.847</td><td>0.771</td><td>0.952</td></tr><tr><td>MoG10 ad50 id50 mtC r0.3</td><td>1.406</td><td>1.407</td><td>1.451</td><td>1.404</td><td>1.487</td><td>1.417</td><td>1.383</td><td>1.282</td><td>1.253</td><td>1.289</td></tr><tr><td>MoG10 ad50 id50 mtC r1.0</td><td>2.512</td><td>2.511</td><td>2.520</td><td>2.508</td><td>2.223</td><td>2.477</td><td>2.267</td><td>2.148</td><td>2.179</td><td>2.271</td></tr><tr><td>MoG10 ad50 id50 mtC r3.0</td><td>4.268</td><td>4.263</td><td>4.256</td><td>4.258</td><td>3.730</td><td>4.188</td><td>3.654</td><td>3.646</td><td>3.644</td><td>3.653</td></tr><tr><td>MoG2 ad50 id50 mtR r0.03</td><td>0.619</td><td>0.749</td><td>2.808</td><td>0.600</td><td>0.758</td><td>1.033</td><td>6.274</td><td>2.061</td><td>0.805</td><td>5.359</td></tr><tr><td>MoG5 ad50 id50 mtR r0.03</td><td>0.791</td><td>0.640</td><td>5.142</td><td>0.595</td><td>1.550</td><td>1.090</td><td>1.949</td><td>2.242</td><td>1.024</td><td>1.866</td></tr><tr><td>MoG10 ad50 id50 mtR r0.03</td><td>0.703</td><td>0.980</td><td>6.302</td><td>0.884</td><td>1.794</td><td>1.475</td><td>2.977</td><td>3.293</td><td>1.352</td><td>3.296</td></tr><tr><td>MoG20 ad50 id50 mtR r0.03</td><td>1.227</td><td>1.251</td><td>6.981</td><td>3.111</td><td>2.558</td><td>2.514</td><td>5.092</td><td>3.657</td><td>1.742</td><td>4.159</td></tr><tr><td>MoG50 ad50 id50 mtR r0.03</td><td>1.642</td><td>1.524</td><td>7.488</td><td>5.298</td><td>3.184</td><td>3.805</td><td>6.255</td><td>4.636</td><td>2.783</td><td>5.328</td></tr><tr><td>MoG10 ad50 id50 mtR r0.1</td><td>1.059</td><td>1.321</td><td>6.241</td><td>1.096</td><td>2.250</td><td>1.944</td><td>2.808</td><td>3.565</td><td>1.552</td><td>2.233</td></tr><tr><td>MoG10 ad50 id50 mtR r0.3</td><td>1.637</td><td>1.876</td><td>6.256</td><td>1.816</td><td>2.573</td><td>2.427</td><td>5.595</td><td>3.043</td><td>2.357</td><td>2.448</td></tr><tr><td>MoG10 ad50 id50 mtR r1.0</td><td>2.771</td><td>2.969</td><td>6.481</td><td>2.861</td><td>3.145</td><td>3.422</td><td>3.989</td><td>3.626</td><td>2.898</td><td>3.718</td></tr><tr><td>MoG10 ad50 id50 mtR r3.0</td><td>4.617</td><td>4.764</td><td>7.254</td><td>4.664</td><td>4.457</td><td>5.114</td><td>4.710</td><td>4.790</td><td>4.331</td><td>4.413</td></tr><tr><td/><td>oracle</td><td>KDE</td><td/><td>Gaussian</td><td/><td>MOG</td><td>VAE</td><td colspan="2">RealNVP</td><td>GAN</td><td>WGAN</td><td>WGAN-GP</td><td>GAN</td></tr><tr><td>MoG2 ad50 id50 mtC r0.03</td><td>73.628</td><td>73.576</td><td/><td>8.073</td><td/><td>73.656</td><td>72.604</td><td/><td>70.636</td><td>84.551</td><td>-18.458</td><td>-65.469</td><td>84</td></tr><tr><td>MoG5 ad50 id50 mtC r0.03</td><td>72.724</td><td>72.700</td><td/><td>17.434</td><td/><td>72.743</td><td>15.856</td><td/><td>62.414</td><td>-38.952</td><td>31.335</td><td>85.773</td><td>81</td></tr><tr><td>MoG10 ad50 id50 mtC r0.03</td><td>72.095</td><td>72.011</td><td/><td>31.006</td><td/><td>72.045</td><td>11.079</td><td/><td>56.194</td><td>1.525</td><td>24.857</td><td>60.130</td><td>89</td></tr><tr><td>MoG20 ad50 id50 mtC r0.03</td><td>71.276</td><td>71.234</td><td/><td>33.946</td><td/><td>70.386</td><td>8.195</td><td/><td>60.910</td><td>81.715</td><td>45.758</td><td>78.979</td><td>80</td></tr><tr><td>MoG50 ad50 id50 mtC r0.03</td><td>70.864</td><td>70.936</td><td/><td>34.300</td><td/><td>70.844</td><td>2.004</td><td/><td>62.431</td><td>85.098</td><td>69.406</td><td>88.844</td><td>83</td></tr><tr><td>MoG10 ad50 id50 mtC r0.1</td><td>42.003</td><td>41.926</td><td/><td>30.298</td><td/><td>41.679</td><td>38.911</td><td/><td>38.265</td><td>56.746</td><td>45.851</td><td>50.840</td><td>48</td></tr><tr><td>MoG10 ad50 id50 mtC r0.3</td><td>14.726</td><td>14.646</td><td/><td>11.334</td><td/><td>14.641</td><td>27.678</td><td/><td>13.503</td><td>19.792</td><td>29.601</td><td>27.557</td><td>26</td></tr><tr><td>MoG10 ad50 id50 mtC r1.0</td><td>-14.799</td><td>-14.891</td><td/><td>-15.546</td><td/><td>-14.922</td><td>7.104</td><td/><td>-13.602</td><td>5.036</td><td>4.401</td><td>1.838</td><td>5</td></tr><tr><td>MoG10 ad50 id50 mtC r3.0</td><td>-41.662</td><td>-41.765</td><td/><td>-41.811</td><td/><td>-41.771</td><td>-17.932</td><td/><td>-40.008</td><td>-21.709</td><td>-23.253</td><td>-22.324</td><td>-22</td></tr><tr><td>MoG2 ad50 id50 mtR r0.03</td><td>73.628</td><td>73.576</td><td/><td>-1725.184</td><td/><td>73.654</td><td>-107.994</td><td/><td>38.340</td><td>-217.560</td><td>-466.513</td><td>11.341</td><td>-14</td></tr><tr><td>MoG5 ad50 id50 mtR r0.03</td><td>72.724</td><td>72.700</td><td/><td>-4787.562</td><td/><td>72.734</td><td>-423.206</td><td/><td>-35.199</td><td>-298.758</td><td>-983.164</td><td>-103.058</td><td>-287</td></tr><tr><td>MoG10 ad50 id50 mtR r0.03</td><td>72.095</td><td>72.011</td><td/><td>-6885.307</td><td/><td>72.051</td><td>-862.853</td><td/><td>-208.406</td><td>22.882</td><td>-2321.356</td><td>-389.596</td><td>-465</td></tr><tr><td>MoG20 ad50 id50 mtR r0.03</td><td>71.263</td><td>71.227</td><td/><td>-8293.821</td><td colspan="2">-1950.515</td><td>-1162.677</td><td/><td>-904.543</td><td>-99.933</td><td>-2659.067</td><td>-306.468</td><td>-472</td></tr><tr><td>MoG50 ad50 id50 mtR r0.03</td><td>70.270</td><td>70.336</td><td/><td>-9223.065</td><td colspan="2">-4879.046</td><td>-1742.290</td><td colspan="2">-2052.524</td><td>-5468.518</td><td>-3669.330</td><td>-702.292</td><td>-534</td></tr><tr><td>MoG10 ad50 id50 mtR r0.1</td><td>41.996</td><td>41.919</td><td/><td>-2059.011</td><td/><td>41.960</td><td>-253.427</td><td/><td>-71.988</td><td>29.173</td><td>-711.750</td><td>-34.268</td><td>16</td></tr><tr><td>MoG10 ad50 id50 mtR r0.3</td><td>14.531</td><td>14.456</td><td/><td>-684.216</td><td/><td>14.491</td><td>-68.693</td><td/><td>-23.838</td><td>-28.428</td><td>-140.817</td><td>-30.060</td><td>16</td></tr><tr><td>MoG10 ad50 id50 mtR r1.0</td><td>-15.569</td><td>-15.642</td><td/><td>-222.121</td><td/><td>-15.611</td><td>-23.774</td><td/><td>-34.471</td><td>-42.190</td><td>-45.174</td><td>-13.998</td><td>-21</td></tr><tr><td>MoG10 ad50 id50 mtR r3.0</td><td>-43.034</td><td>-43.118</td><td/><td>-110.474</td><td/><td>-43.072</td><td>-31.757</td><td/><td>-50.787</td><td>-34.440</td><td>-41.842</td><td>-31.362</td><td>-32</td></tr></table>

Mixture distributions with 10000 training points using log-likelihood metric.

Table content: <table><tr><td/><td>oracle</td><td>KDE</td><td>Gaussian</td><td>MOG</td><td>VAE</td><td>RealNVP</td><td>GAN</td><td>WGAN</td><td>WGAN-GP</td><td>GAN-NR</td></tr><tr><td>MoG2 ad50 id5 mtC r0.03</td><td>0.237</td><td>0.258</td><td>57.920</td><td>0.233</td><td>26.330</td><td>6.417</td><td>0.627</td><td>22.517</td><td>7.451</td><td>5.940</td></tr><tr><td>MoG5 ad50 id5 mtC r0.03</td><td>0.287</td><td>0.318</td><td>44.034</td><td>0.294</td><td>56.186</td><td>8.342</td><td>4.037</td><td>31.937</td><td>5.233</td><td>3.522</td></tr><tr><td>MoG10 ad50 id5 mtC r0.03</td><td>0.361</td><td>0.391</td><td>36.817</td><td>0.368</td><td>44.635</td><td>9.226</td><td>2.561</td><td>16.402</td><td>3.961</td><td>3.021</td></tr><tr><td>MoG20 ad50 id5 mtC r0.03</td><td>0.446</td><td>0.484</td><td>35.249</td><td>2.085</td><td>46.903</td><td>13.206</td><td>3.322</td><td>87.685</td><td>5.983</td><td>2.621</td></tr><tr><td>MoG50 ad50 id5 mtC r0.03</td><td>0.612</td><td>0.653</td><td>35.234</td><td>2.084</td><td>44.958</td><td>13.368</td><td>6.592</td><td>21.481</td><td>3.420</td><td>3.515</td></tr><tr><td>MoG10 ad50 id5 mtC r0.1</td><td>1.112</td><td>1.208</td><td>38.380</td><td>1.108</td><td>43.889</td><td>245.109</td><td>6.054</td><td>22.469</td><td>4.014</td><td>6.244</td></tr><tr><td>MoG10 ad50 id5 mtC r0.3</td><td>3.256</td><td>3.541</td><td>46.841</td><td>7.086</td><td>50.803</td><td>13.608</td><td>14.102</td><td>36.612</td><td>7.781</td><td>5.641</td></tr><tr><td>MoG10 ad50 id5 mtC r1.0</td><td>10.756</td><td>11.707</td><td>85.637</td><td>10.767</td><td>80.277</td><td>27.669</td><td>16.099</td><td>69.350</td><td>17.915</td><td>14.710</td></tr><tr><td>MoG10 ad50 id5 mtC r3.0</td><td>32.184</td><td>35.043</td><td>205.049</td><td>53.285</td><td>179.895</td><td>51.730</td><td>56.791</td><td>140.923</td><td>47.643</td><td>42.885</td></tr><tr><td>MoG2 ad50 id5 mtR r0.03</td><td>0.237</td><td>0.258</td><td>1688.426</td><td>0.236</td><td>382.969</td><td>211.919</td><td>7457.907</td><td>4194.667</td><td>246.576</td><td>7166.328</td></tr><tr><td>MoG5 ad50 id5 mtR r0.03</td><td>0.287</td><td>0.318</td><td>3340.899</td><td>0.295</td><td>712.163</td><td>333.198</td><td>418.922</td><td>4288.632</td><td>158.407</td><td>404.435</td></tr><tr><td>MoG10 ad50 id5 mtR r0.03</td><td>0.361</td><td>0.391</td><td>4480.731</td><td>0.368</td><td>951.298</td><td>705.150</td><td>1976.482</td><td>1415.009</td><td>373.751</td><td>4292.463</td></tr><tr><td>MoG20 ad50 id5 mtR r0.03</td><td>0.446</td><td>0.484</td><td>5498.846</td><td>1734.484</td><td>1204.290</td><td>884.249</td><td>4191.913</td><td>1494.503</td><td>688.596</td><td>2085.471</td></tr><tr><td>MoG50 ad50 id5 mtR r0.03</td><td>0.612</td><td>0.653</td><td>6568.851</td><td>3224.388</td><td>1464.644</td><td>3181.403</td><td>4096.990</td><td>5522.390</td><td>764.674</td><td>3614.357</td></tr><tr><td>MoG10 ad50 id5 mtR r0.1</td><td>1.112</td><td>1.208</td><td>4436.725</td><td>1.118</td><td>852.832</td><td>725.129</td><td>1964.068</td><td>3033.507</td><td>482.750</td><td>2128.474</td></tr><tr><td>MoG10 ad50 id5 mtR r0.3</td><td>3.256</td><td>3.541</td><td>4389.701</td><td>3.249</td><td>851.563</td><td>722.119</td><td>1948.733</td><td>1964.690</td><td>424.794</td><td>3194.469</td></tr><tr><td>MoG10 ad50 id5 mtR r1.0</td><td>10.756</td><td>11.707</td><td>4326.580</td><td>10.872</td><td>981.120</td><td>710.919</td><td>2978.250</td><td>1837.948</td><td>347.022</td><td>709.904</td></tr><tr><td>MoG10 ad50 id5 mtR r3.0</td><td>32.184</td><td>35.043</td><td>4341.083</td><td>32.344</td><td>1002.917</td><td>716.591</td><td>2654.381</td><td>1298.073</td><td>389.408</td><td>2183.150</td></tr><tr><td>MoG2 ad50 id50 mtC</td><td>20.799</td><td>21.094</td><td>62.118</td><td>20.821</td><td>27.443</td><td>22.756</td><td>18.180</td><td>88.884</td><td>125.598</td><td>17.226</td></tr><tr><td>MoG5 ad50 id50 mtC r0.03</td><td>21.699</td><td>22.049</td><td>51.226</td><td>21.732</td><td>46.620</td><td>28.345</td><td>148.114</td><td>48.010</td><td>20.331</td><td>19.458</td></tr><tr><td>MoG10 ad50 id50 mtC r0.03</td><td>22.485</td><td>22.895</td><td>45.508</td><td>22.514</td><td>53.014</td><td>30.658</td><td>66.959</td><td>49.907</td><td>32.291</td><td>17.079</td></tr><tr><td>MoG20 ad50 id50 mtC r0.03</td><td>23.445</td><td>23.845</td><td>44.976</td><td>23.861</td><td>53.748</td><td>29.253</td><td>20.751</td><td>36.795</td><td>22.505</td><td>21.106</td></tr><tr><td>MoG50 ad50 id50 mtC r0.03</td><td>23.781</td><td>24.107</td><td>44.580</td><td>23.829</td><td>55.255</td><td>28.791</td><td>20.028</td><td>25.153</td><td>17.247</td><td>20.378</td></tr><tr><td>MoG10 ad50 id50 mtC r0.1</td><td>74.894</td><td>76.283</td><td>92.325</td><td>75.530</td><td>82.746</td><td>80.491</td><td>59.927</td><td>71.082</td><td>63.669</td><td>67.298</td></tr><tr><td>MoG10 ad50 id50 mtC r0.3</td><td>222.124</td><td>225.794</td><td>235.964</td><td>222.485</td><td>220.458</td><td>227.260</td><td>205.072</td><td>171.029</td><td>174.666</td><td>178.685</td></tr><tr><td>MoG10 ad50 id50 mtC r1.0</td><td>719.093</td><td>730.723</td><td>730.422</td><td>720.277</td><td>517.921</td><td>701.075</td><td>560.057</td><td>513.860</td><td>529.873</td><td>562.953</td></tr><tr><td>MoG10 ad50 id50 mtC r3.0</td><td>2099.189</td><td>2131.176</td><td>2106.525</td><td>2099.680</td><td>1518.626</td><td>2027.237</td><td>1496.654</td><td>1497.029</td><td>1483.389</td><td>1498.844</td></tr><tr><td>MoG2 ad50 id50 mtR r0.03</td><td>20.804</td><td>21.094</td><td>1630.390</td><td>20.810</td><td>411.953</td><td>149.912</td><td>6920.808</td><td>362.710</td><td>195.030</td><td>6064.530</td></tr><tr><td>MoG5 ad50 id50 mtR r0.03</td><td>21.702</td><td>22.051</td><td>3232.852</td><td>21.696</td><td>658.708</td><td>246.329</td><td>434.970</td><td>1011.023</td><td>341.312</td><td>430.872</td></tr><tr><td>MoG10 ad50 id50 mtR r0.03</td><td>22.487</td><td>22.896</td><td>4356.962</td><td>22.518</td><td>953.083</td><td>458.777</td><td>2980.873</td><td>2383.723</td><td>570.546</td><td>651.277</td></tr><tr><td>MoG20 ad50 id50 mtR r0.03</td><td>23.494</td><td>23.890</td><td>5377.364</td><td>1811.709</td><td>1113.193</td><td>1019.912</td><td>4360.021</td><td>2203.143</td><td>512.071</td><td>2302.820</td></tr><tr><td>MoG50 ad50 id50 mtR r0.03</td><td>25.069</td><td>25.495</td><td>6471.202</td><td>3583.002</td><td>1405.041</td><td>1671.400</td><td>4350.273</td><td>2992.306</td><td>1376.382</td><td>3516.809</td></tr><tr><td>MoG10 ad50 id50 mtR r0.1</td><td>74.937</td><td>76.321</td><td>4259.335</td><td>75.057</td><td>1001.369</td><td>496.565</td><td>3162.742</td><td>2475.862</td><td>471.726</td><td>138.372</td></tr><tr><td>MoG10 ad50 id50 mtR r0.3</td><td>224.793</td><td>228.979</td><td>4181.012</td><td>225.307</td><td>961.619</td><td>584.302</td><td>4800.571</td><td>1403.601</td><td>694.724</td><td>241.683</td></tr><tr><td>MoG10 ad50 id50 mtR r1.0</td><td>749.292</td><td>763.307</td><td>4351.709</td><td>750.725</td><td>1059.480</td><td>1100.161</td><td>1375.055</td><td>1479.128</td><td>827.537</td><td>1005.823</td></tr><tr><td>MoG10 ad50 id50 mtR r3.0</td><td>2247.890</td><td>2288.899</td><td>5581.006</td><td>2251.496</td><td>1971.971</td><td>2580.506</td><td>2141.336</td><td>2282.581</td><td>1894.595</td><td>1986.461</td></tr></table>

Mixture distributions with 10000 training points using Hausdorff metric.

Table content: <table><tr><td/><td>oracle</td><td>KDE</td><td>Gaussian</td><td>MOG</td><td>VAE</td><td>RealNVP</td><td>GAN</td><td>WGAN</td><td>WGAN-GP</td><td>GAN-NR</td></tr><tr><td>MoG2 ad50 id5 mtC r0.03</td><td>2.00</td><td>2.00</td><td>2.00</td><td>2.00</td><td>2.00</td><td>1.99</td><td>2.00</td><td>1.99</td><td>2.00</td><td>2.00</td></tr><tr><td>MoG5 ad50 id5 mtC r0.03</td><td>5.00</td><td>5.00</td><td>4.99</td><td>5.00</td><td>4.90</td><td>4.93</td><td>4.53</td><td>4.92</td><td>4.99</td><td>4.99</td></tr><tr><td>MoG10 ad50 id5 mtC r0.03</td><td>9.99</td><td>9.98</td><td>9.92</td><td>9.98</td><td>9.39</td><td>9.76</td><td>9.86</td><td>9.91</td><td>9.98</td><td>9.98</td></tr><tr><td>MoG20 ad50 id5 mtC r0.03</td><td>19.96</td><td>19.91</td><td>18.94</td><td>19.75</td><td>12.21</td><td>18.84</td><td>19.76</td><td>8.13</td><td>19.83</td><td>19.91</td></tr><tr><td>MoG50 ad50 id5 mtC r0.03</td><td>49.65</td><td>49.75</td><td>41.52</td><td>47.61</td><td>18.71</td><td>46.89</td><td>49.41</td><td>42.50</td><td>49.46</td><td>46.40</td></tr><tr><td>MoG10 ad50 id5 mtC r0.1</td><td>9.99</td><td>9.98</td><td>9.92</td><td>9.98</td><td>9.51</td><td>9.69</td><td>9.93</td><td>9.43</td><td>9.98</td><td>9.92</td></tr><tr><td>MoG10 ad50 id5 mtC r0.3</td><td>9.99</td><td>9.98</td><td>9.91</td><td>9.98</td><td>9.44</td><td>9.81</td><td>9.74</td><td>9.73</td><td>9.98</td><td>9.99</td></tr><tr><td>MoG10 ad50 id5 mtC r1.0</td><td>9.99</td><td>9.98</td><td>9.94</td><td>9.99</td><td>9.20</td><td>9.78</td><td>9.93</td><td>9.79</td><td>9.98</td><td>9.99</td></tr><tr><td>MoG10 ad50 id5 mtC r3.0</td><td>9.99</td><td>9.98</td><td>9.97</td><td>9.99</td><td>9.43</td><td>9.73</td><td>9.05</td><td>9.95</td><td>9.94</td><td>9.95</td></tr><tr><td>MoG2 ad50 id5 mtR r0.03</td><td>2.00</td><td>2.00</td><td>2.00</td><td>2.00</td><td>2.00</td><td>2.00</td><td>1.00</td><td>1.10</td><td>2.00</td><td>1.00</td></tr><tr><td>MoG5 ad50 id5 mtR r0.03</td><td>5.00</td><td>5.00</td><td>4.99</td><td>5.00</td><td>5.00</td><td>4.96</td><td>4.97</td><td>4.81</td><td>4.95</td><td>4.97</td></tr><tr><td>MoG10 ad50 id5 mtR r0.03</td><td>9.99</td><td>9.98</td><td>9.87</td><td>9.99</td><td>9.88</td><td>9.84</td><td>8.74</td><td>9.97</td><td>9.97</td><td>5.73</td></tr><tr><td>MoG20 ad50 id5 mtR r0.03</td><td>19.96</td><td>19.91</td><td>19.32</td><td>19.94</td><td>19.44</td><td>18.80</td><td>9.38</td><td>19.41</td><td>19.71</td><td>14.91</td></tr><tr><td>MoG50 ad50 id5 mtR r0.03</td><td>49.65</td><td>49.75</td><td>41.79</td><td>49.44</td><td>47.06</td><td>46.01</td><td>17.26</td><td>45.32</td><td>48.64</td><td>15.49</td></tr><tr><td>MoG10 ad50 id5 mtR r0.1</td><td>9.99</td><td>9.98</td><td>9.88</td><td>9.99</td><td>9.76</td><td>9.84</td><td>8.73</td><td>9.67</td><td>9.99</td><td>6.86</td></tr><tr><td>MoG10 ad50 id5 mtR r0.3</td><td>9.99</td><td>9.98</td><td>9.88</td><td>9.99</td><td>9.93</td><td>9.87</td><td>8.69</td><td>9.88</td><td>9.89</td><td>7.58</td></tr><tr><td>MoG10 ad50 id5 mtR r1.0</td><td>9.99</td><td>9.98</td><td>9.87</td><td>9.98</td><td>9.91</td><td>9.30</td><td>7.79</td><td>9.92</td><td>9.99</td><td>9.19</td></tr><tr><td>MoG10 ad50 id5 mtR r3.0</td><td>9.99</td><td>9.98</td><td>9.87</td><td>9.97</td><td>9.87</td><td>9.51</td><td>7.84</td><td>9.93</td><td>9.94</td><td>7.46</td></tr><tr><td>MoG2 ad50 id50 mtC r0.03</td><td>2.00</td><td>2.00</td><td>2.00</td><td>2.00</td><td>2.00</td><td>1.99</td><td>1.40</td><td>1.87</td><td>1.99</td><td>1.98</td></tr><tr><td>MoG5 ad50 id50 mtC r0.03</td><td>5.00</td><td>5.00</td><td>5.00</td><td>5.00</td><td>4.93</td><td>4.97</td><td>4.79</td><td>4.97</td><td>5.00</td><td>4.99</td></tr><tr><td>MoG10 ad50 id50 mtC r0.03</td><td>9.99</td><td>9.98</td><td>9.99</td><td>9.98</td><td>8.99</td><td>9.87</td><td>9.70</td><td>9.87</td><td>8.07</td><td>9.92</td></tr><tr><td>MoG20 ad50 id50 mtC r0.03</td><td>19.96</td><td>19.90</td><td>19.97</td><td>19.91</td><td>18.89</td><td>19.69</td><td>19.54</td><td>19.94</td><td>19.72</td><td>19.87</td></tr><tr><td>MoG50 ad50 id50 mtC r0.03</td><td>49.77</td><td>49.68</td><td>49.82</td><td>49.56</td><td>47.83</td><td>49.47</td><td>48.79</td><td>49.64</td><td>46.83</td><td>49.77</td></tr><tr><td>MoG10 ad50 id50 mtC r0.1</td><td>9.99</td><td>9.98</td><td>9.99</td><td>9.97</td><td>9.47</td><td>9.94</td><td>9.44</td><td>9.81</td><td>9.92</td><td>7.09</td></tr><tr><td>MoG10 ad50 id50 mtC r0.3</td><td>9.99</td><td>9.98</td><td>9.99</td><td>9.98</td><td>5.13</td><td>9.88</td><td>9.63</td><td>9.31</td><td>9.95</td><td>9.65</td></tr><tr><td>MoG10 ad50 id50 mtC r1.0</td><td>9.99</td><td>9.99</td><td>9.99</td><td>9.98</td><td>9.93</td><td>9.98</td><td>9.17</td><td>9.97</td><td>9.67</td><td>9.37</td></tr><tr><td>MoG10 ad50 id50 mtC r3.0</td><td>9.99</td><td>9.99</td><td>10.00</td><td>9.99</td><td>9.94</td><td>9.98</td><td>9.94</td><td>9.28</td><td>9.85</td><td>9.95</td></tr><tr><td>MoG2 ad50 id50 mtR r0.03</td><td>2.00</td><td>2.00</td><td>2.00</td><td>2.00</td><td>2.00</td><td>2.00</td><td>1.00</td><td>1.99</td><td>2.00</td><td>1.00</td></tr><tr><td>MoG5 ad50 id50 mtR r0.03</td><td>5.00</td><td>5.00</td><td>5.00</td><td>5.00</td><td>4.92</td><td>4.97</td><td>4.74</td><td>4.99</td><td>5.00</td><td>4.77</td></tr><tr><td>MoG10 ad50 id50 mtR r0.03</td><td>9.99</td><td>9.98</td><td>9.95</td><td>9.98</td><td>9.96</td><td>9.94</td><td>7.43</td><td>9.78</td><td>9.99</td><td>8.33</td></tr><tr><td>MoG20 ad50 id50 mtR r0.03</td><td>19.96</td><td>19.91</td><td>19.31</td><td>19.93</td><td>19.24</td><td>18.75</td><td>8.50</td><td>19.87</td><td>19.45</td><td>13.30</td></tr><tr><td>MoG50 ad50 id50 mtR r0.03</td><td>49.65</td><td>49.75</td><td>40.94</td><td>48.41</td><td>46.33</td><td>36.63</td><td>26.17</td><td>47.89</td><td>44.29</td><td>21.10</td></tr><tr><td>MoG10 ad50 id50 mtR r0.1</td><td>9.99</td><td>9.98</td><td>9.95</td><td>9.99</td><td>9.90</td><td>9.87</td><td>7.81</td><td>9.87</td><td>9.97</td><td>9.37</td></tr><tr><td>MoG10 ad50 id50 mtR r0.3</td><td>9.99</td><td>9.98</td><td>9.95</td><td>9.98</td><td>9.86</td><td>9.86</td><td>4.29</td><td>9.96</td><td>9.87</td><td>9.53</td></tr><tr><td>MoG10 ad50 id50 mtR r1.0</td><td>9.99</td><td>9.98</td><td>9.96</td><td>9.99</td><td>9.96</td><td>9.78</td><td>9.08</td><td>9.85</td><td>9.97</td><td>9.76</td></tr><tr><td>MoG10 ad50 id50 mtR r3.0</td><td>9.99</td><td>9.98</td><td>9.95</td><td>9.98</td><td>9.91</td><td>9.76</td><td>9.82</td><td>9.92</td><td>9.98</td><td>9.95</td></tr></table>

Mixture distributions with 10000 training points using mode-coverage metric.

Table content: <table/>

Peer review (iclr2019):
 Overall, this is a thorough attempt at a system for evaluating various generative models on synthetic problems vaguely representative of the kinds of problems claimed to be covered by GANs. I think the approach and the conclusions drawn are mostly reasonable, with one major caveat discussed shortly. I also think it would help in a revision to add evaluations of more recent successors to RealNVP, such as MAF (NIPS 2017, https://arxiv.org/abs/1705.07057 ), Glow ( https://arxiv.org/abs/1807.03039 ), and (although of course this paper came out concurrently with your submission) the promising FFJORD ( https://openreview.net/forum?id=rJxgknCcK7 ). The scale of comparison of GAN variants is also much smaller than that of Lucic et al. or their followup, Kurach et al. ( https://arxiv.org/abs/1807.04720 ), which is not cited here (and should be). But primarily, I think there are some serious concerns with your choice of metrics that make the results as they are difficult to interpret. "Note that OT is not a distance in the standard mathematical sense, as for instance the *distance* between two sets of points sampled from the same distribution is not zero." -- You*ve confused some notions here. The Wasserstein-1 distance, which is a scalar times the variant of OT you use here, absolutely is a proper distance metric between distributions: W(P, Q) is a metric. But when you compute the OT distance between *samples*, OT(S, T) with S ~ P and T ~ Q, you*re equivalently computing the distance W(.hat{P}, .hat{Q}) between the empirical distributions of the samples, .hat{P} = 1/N .sum_i .delta_{S_i} and the similar .hat{Q}, which of course are not the same thing as the source distributions themselves. You can, though, view OT(S, T) as an *estimator* of W(P, Q); the distance between *distributions* is what we actually care about. It is well-known that these empirical distributions of samples .hat{P} converge to the true distribution P (in the Wasserstein sense, W(P, .hat{P})) exponentially slowly in the dimension, which is what your example about high-dimensional distributions demonstrates. Incidentally, this is exactly the example used in Arora et al. (ICML 2017, https://arxiv.org/abs/1703.00573 ). This means that, viewed as an estimator of the true distance between distributions, the empirical-distribution OT estimator is strongly biased. Thus it becomes very difficult to tell what the true OT value is at any sample size, and moreover this amount of bias might differ for different distribution pairs even at the same sample size, so *comparing* OT estimates at a fixed sample size is a tricky business. For example, in your Figure 2, when the "oracle" score is significantly more than zero, you know that all of your estimates are very strongly biased. There is not, as far as I know, any strong reason to suspect that this amount of bias should be comparable for different distribution pairs, making any conclusions drawn from these numbers suspect. Your scheme you call "Two-Sample Test," first, should have a more specific name. Two-sample testing is an extremely broad field, with instances including the classical Kolmogorov-Smirnov test and t tests, the popular-in-ML kernel MMD-based tests, and even Wasserstein-based tests (e.g. https://arxiv.org/abs/1509.02237 ). Previous applications of these tests in GANs and generative models include Bounliphone et al. (ICLR 2016, https://arxiv.org/abs/1511.04581 ), Lopez-Paz and Oquab (2016 - which you cite without a venue but which was at ICLR 2017), Sutherland et al. (ICLR 2017, https://arxiv.org/abs/1611.04488 ), Huang et al. (2018), and more, using a variety of schemes. Your name for this should include "nearest neighbor" or something along those lines to avoid confusion. Also, you call this an "extension of the original formulation," but in the common case where n(x) is more often right than wrong, your v is exactly .hat t - 1 of Lopez-Paz and Oquab; see their (2). If it*s usually wrong, then v = 1 - .hat t; only when the signs differ per class does it significantly differ from theirs, and in any case I don*t see a real motivation to put the absolute values for each class separately rather than just taking |.hat t - 1/2|. Moreover, it*s kind of crazy to term your v statistic a two-sample *test* -- you have nothing in there about its sampling distribution, which is key to hypothesis testing to obtain e.g. a p-value. (Maybe the variance of v is very different between different distributions; this is likely the case. In any case the variance will probably become extremely large as the dimension increases.) Comparing this score is thus difficult, but in any case calling it a "test" is potentially very misleading. You could, though, estimate the variance as described by Lopez-Paz and Oquab to construct a test. Also: you can imagine the statistic v(S, T) as an estimator of the distance between distributions given as D(P, Q) = |1/2 - .int ( 1 if p(x) > q(x), 0 o.w.) p(x) dx| + |1/2 - .int (-1 if p(x) > q(x), 0 o.w.) q(x) dx|. But v(S, T) is, like for the OT distance, a biased estimator of this distance, whose bias will get worse with the dimension. Thus, like with the OT, it*s hard to meaningfully compare v(S, T) as an attempt to compare *distributions* based on D, which is what we actually care about. Here the oracle score does not show strong bias: assuming a reasonable number of samples, when P = Q the v estimator is always going be approximately 0. But this doesn*t mean that other estimators aren*t strongly biased, and indeed this is exactly what your Appendix C shows. The strong change in performance for KDE is somewhat hard to interpret, but maybe has something to do with the connection between KDE and NN-based methods? Your log-likelihood score is an unbiased and asymptotically normal estimate of the true distribution score (the cross-entropy), so it*s easy to compare. But it accounts only for a very small portion of comparing distributions. There is at least one score in common use for this kind of evaluation with easy-to-compare estimators: the squared MMD. It has an easy-to-compute unbiased and asymptotically normal estimator, so it*s easy to get confidence intervals for the true value between distributions at any sample size, making comparing the numbers based on a reasonable number of samples easy. There*s also a well-devolped theory for how to construct p-values for a test if you want those; Bounliphone et al. above even developed a relative test to compare the MMDs of two models accounting for the correlations due to using the same "target" set, though if you use separate target sets (because you can easily sample more points from your synthetic distribution) then it*s simpler. The choice of kernel does matter, but I think the median-heuristic Gaussian kernel would be a very reasonable score to add to your repertoire, and for particular distributions you also might be able to pick a better kernel (e.g. based on the causal factors when those exist). See also Binkowski et al. (ICLR 2018, https://arxiv.org/abs/1801.01401 ) for a detailed discussion of these issues in comparison to the FID score. Using a metric whose estimation can be understood, and whose estimators can be reliably compared, is I think vital to any evaluation process. This also prevents issues like when RealNVP outperforms the oracle, which should be impossible with any proper evaluation metric. Minor points: - Why is Pedregosa et al. (2011) cited for fitting multivariate Gaussians by maximum likelihood? This is something that doesn*t need a citation, especially not to scikit-learn, which doesn*t even (I don*t think) contain an implementation of fitting Gaussians beyond (np.mean(X, axis=0), np.cov(X, rowvar=False)). - Mode coverage and related scores: this is based on assigning sample points to their single most likely clusters? I*d imagine that sometimes a model will output points far from any cluster, in which case the cluster that happens to be closest might happen to be the most likely, but it*s strange to really count that point as part of that cluster for these scores. Or similarly, a point might be relatively evenly spaced between two clusters, in which case the assignment could be fairly arbitrary, again making these scores a little strange.