A GOODNESS OF FIT MEASURE FOR GENERATIVE NETWORKS
Authors: Lorenzo Luzi
Randall Balestriero
Richard Baraniuk
Venue:  (2020)
Keywords: generative adversarial networks
goodness of fit
inception score
empirical approximation error
validation metric
frechet inception score

Abstract: We define a goodness of fit measure for generative networks which captures how well the network can generate the training data, which is necessary to learn the true data distribution.
We demonstrate how our measure can be leveraged to understand mode collapse in generative adversarial networks and provide practitioners with a novel way to perform model comparison and early stopping without having to access another trained model as with Frechet Inception Distance or Inception Score. This measure shows that several successful, popular generative models, such as DCGAN and WGAN, fall very short of learning the data distribution. We identify this issue in generative models and empirically show that overparameterization via subsampling data and using a mixture of models improves performance in terms of goodness of fit.

Under review as a conference paper at ICLR 2020
A GOODNESS OF FIT MEASURE FOR GENERATIVE NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We define a goodness of fit measure for generative networks which captures how well the network can generate the training data, which is necessary to learn the true data distribution. We demonstrate how our measure can be leveraged to under- stand mode collapse generative adversarial networks and provide practitioners with novel way to perform model comparison and early stopping without having to access another trained model as with Frechet Inception Distance or Inception Score. This measure shows that several successful, popular generative models, such as DCGAN and WGAN, fall very short of learning the data distribution. We identify this issue in generative models and empirically show that overparameteri- zation via subsampling data and using a mixture of models improves performance in terms of goodness of fit.
INTRODUCTION AND RELATED WORK
Generative adversarial networks (Goodfellow et al. 2014) are specific type of generative model that has shown impressive performance lately. The main idea is that there are two networks that compete against each other: a generator network that generates images and discriminator network that tries to distinguish between real and fake images. These models are useful because they can generate very realistic images that are not in the training set. Throughout the rest of the paper, we will use GANs as a specific class models to study, however, the goodness of measure discussed in Section 3.2 and its applications can be extended to other generative networks such as Variational Autoencoders.
Some GANs that appear be successful in practice cannot actually reproduce the training set, as we will see. Other generative models, such as Generative Latent Optimization (GLO) (Bojanowski al. 2017) and Implicit Maximum Likelihood Estimation (IMLE) (Li and Malik 2018) attempt to memorize the training data as part of the learning algorithm. These methods are not as successful as GANs in producing realistic images. We believe that the reason for this difference in performance is due to a lack of overparameterizztion in GANs, GLO, and IMLE.
Our solution starts with measuring how well generative model can generate the training data. We explain in Section 3.2 that our goodness of fit measure F(G) is zero if we are able to perfectly generate the training data. we cannot generate the training data, then F(G) represents how far away we are from generating our training set in an average of total least square sense. We use this goodness of fit measure to evaluate different models and training settings as well as study the evolution of the approximation error through training.
Ideally, we would like overparameterize GANs in order to increase their capacity and reduce F(G). Recently, it was shown that overparameterizztioon in classifiers and autoencoders leads to better performance (Radhakrishnan et al. 2019 Belkin al. 2018). Another reason to overparam- eterize is that observed, while calculating F. that models actually use the the full potential of the latent distribution on z to generate different images. In other words, suppose that we train a GAN with z (0, I), then observe that the optimal z's corresponding ated images from the training set are also distributed as (0, I). That is, once trained, the latent distribution fixed a priori becomes the optimal one minimizing the approximation error.
the closest gener-
Despite the above findings, increasing the complexity of the generator becomes very difficult due to the training algorithms for GANs requiring careful hyper-parameter settings for convergence (Rege

---

Under review as a conference paper at ICLR 2020
and Monteleoni 2019). As such, we explore two alternatives that do not impact the training stability. First, increasing the dimension of the latent space which is currently set to 100 across models and dataset. We demonstrate that this solution indeed allows to reduce the approximation error. Second, we consider a mixture of GAN setting. That is, we train K different GANs on subsets of the data of size approximately N for total data size N. Note that training mixture of GANs has been done in practice Hoang et al. (2017), we thus will quantify the approximation еπтоr reduction that can be obtained with this solution. Hence, we see that if our original GAN has P parameters, now have K parameters total. Also, each one is trained on N/K data. Hence, we see that our N K2 P. N. Hence can get effective overparameterization of 100 fold if we divide our data into K = 10 subsets. We first demonstrate how single GAN trained on smaller dataset has a smaller approximation error. In particular also find that how the dataset is subsampled, random or from clustering based partitioning, matters for performances. We then build on those finding train the mixture of GANs on a K-means based partitioned dataset and demonstrate important reduction in the approximation error.
"effeective" parameters for this mixture of GANs is P
We summarize our contributions that apply to arbitrary generative models s.a. GANs, VAE as fol- lows:
We provide a novel goodness of fit measure for generative networks and how defines necessary conditions for generative networks to be optimal (Sec. 3.II. We also relate the metric to mode collapse and provide implementation details on computing it efficiently (Sec. 3.2) Finally we demonstrate how our metric compares standard GAN metric s.a. the Frechet score (Sec. 3.3)
We demonstrate how our goodness of fit metric allows to gain novel insights into GANs. We show that DCGAN and WGAN do not memorize and have very different behavior w.r.t. overfit- ting Sec. 4.1 In particular DCGAN is able match with WGAN performances if early stopping is performed. We then show the impact architecture and residual connection (Sec. 4.2) Finally, we study the latent space distribution and in particular demonstrate how the optimal latent space distribution which minimizes the approximation error of a trained GAN matches the distribution used for training, highlighting how current approximation errors are due to underparametrized GANs (Sec. 4.3
We provide two solutions to reduce the approximation error without altering training stability Sec. [5] First, we propose increasing the latent space dimension in Sec. [.] Then study
how dataset subsampling also helps reducing the approximation error Sec. 5.2 which motives the use of mixture of GANs Sec. 5.3
2 BACKGROUND
In this section briefly overview Generative Adversarial Networks (GANs) and generative latent optimization (GLO) (Bojanowski et al. 2017). which is another generative model. Finally describe how optimize the generator network latent space to obtain desired generated sample. We remind that all our development applies to arbitrary black/white box generative networks even though we only focus GANs in this paper.
Generative Adversarial Networks. Generative adversarial networks (GANs) are generative neural networks that an adversarial loss; the adversarial loss is typically another neural network. In other words, a GAN consists of two neural networks that compete against each other. The generator network G E Rº generates p-dimensional images from an Ê-dimensional latent space. The discriminator D R + (0, 1) is a classifier which is trained to distinguish between the training set and generated images. The training loss for a batch size of Ng for the discriminator is given by
1 Ng
NB
1
CD
log(D(z.))
log(1 D(G(z,)))
NB
NB
where æ; and G(z,) ) is a real image and a generated image for each E 21,2,.... Nn). respectively. The generator loss is given by
1 Σ
CD
log(1 - D(G
NB
2

---

Under review as a conference paper at ICLR 2020
Notice that the loss for the generator does not explicitly the training data. Instead, the training data is used indirectly through the training of the discriminator.
In this paper, discuss two popular GANs; DCGAN (Radford et al. 2015) which uses the loss above and WGAN (Arjovsky et al. 2017) which uses a slightly different learning algorithm.
Generative latent optimization. In contrast to GANs, GLO is generative network that does not use an adversarial loss. Instead, GLO attempts to memorize the training data by using this loss:
Na _ min ((G(2),
La = .N
SER
where L is a loss function. the original paper, the authors use different loss functions demon- strate how the model differs.
Latent space optimization. The generator a mapping from latent space R into an image space RP. In the GLO paper one aims at finding a specific 2 E IR' such that the generated sample G(z) is close to a target output. In particular, one picks a target as being a randomly generated GAN image from some target vector z". That is, the target is guaranteed to lie in the span of G. Following this, one aims at finding the vector that led to the target G(z*) by
arg min [[((z) - G(
ZER
Since the above optimization problem non-convex, there is not theoretic guarantee finding a global minima, in general. However, empirically it was shown that the above optimization problem is solved 100% of the time in practice (Lipton and Tripathi 2017). We now propose to leverage the above to develop goodness of fit measure.
GOODNESS OF FIT METRIC
In this section we first motivate and define our metric (Sec. 3.1 and provide its approximation 3.2). We demonstrate that our measure being minimized is a necessary and sufficient condition to detect mode collapse. Finally, we study the different with current GAN measures that the Inception Score and the Frechet Inception Score (Sec. 3.3
3.1 METRIC, OPTIMUM GENERATIVE NETWORK, AND MODE COLLAPSE
The generator G is continuous mapping for any type of layer used as current layers in deep learning are all continuous. As such, the following defines the image of the generator:
(G) = (G(F): : F RR).
with the dimension of the latent space. Then the approximation of the true data manifold denoted as X' with G can be measure by "how far" is the span of G. Since the two quantities to compare are sets one solution is the following standard Total Least Square metric defines as
d dlImag(G),x)
min [[G(z) - ær|dæ
(3)
which in practice with finite dataset becomes our proposed measure:
1 Σ min [[G(=) F F(G,X) N
Σ m(x G).
As result, F is an empirical average least square distance between reference points, the observed input, and Ima
Turning the above argument into a probabilistic setting, we obtain the following motivation. A gen- erative model or network learned to approximate some target distribution. This target distribution is itself observed via some samples, the given observations. It is common to use the likelihood measure of fitness for such models, and it is defined L(X) I.w p(#n), with p(x..)
3

---

Under review as a conference paper at ICLR 2020
some distribution density. A necessary condition to allow maximization of the likelihood is that the samples lie in the support of the distribution, that is p(xx,, > 0 3 In E Support(p), as Support(p) {or X p(a) > 0}. In the case of generative network, the distribution p(z) is not easily available. However its support is directly accessible as we have
Proposition 1. Support(p) Imag(G).
As result, from probabilistic fitting point of view, ensuring that all the samples lie in the span of the generator is a necessary condition that must be fulfilled, which otherwise would prevent maximization of the likelihood, leading the following result.
Theorem 1. The optimal generative model in term of distribution approximation must have F(G,X)= = O.
The above setting also allows the following direct and intuitive result that any sample in X' such that m[æ; G) = 0 is a sample that can be generated, potentially with very small probability, by generative network. We conveniently employed the La our metric and we will use this throughout the rest of the paper. However, notice that if X' C Imag( G),, the choice of the distance function is not important because d(x:, y) implies that æ y for any distance metric d.
Relation to mode collapse. We now relate the value of m(x G) for samples from the training set as memorized sample and in general
to mode collapse. We also refer to the case m(x G) we refer as memorization F(X, G) 0. The lack of memorization in current generative networks is evident when generators experience mode collapse. Mode collapse translates into the absence of approximation of the generator some parts of the target distribution. Suppose that the data comes from a distribution Px. Then, mode collapse happens Px (x) > but min, ||G(z) - x|| > In practice, we do not have Px, but we do have the empirical distribution Px, so that mode collapse will occur if we do not memorize the training data. A generative model must memorize the training data in order to avoid mode collapse. We thus obtain the following result.
Proposition 2. A necessary condition avoid mode collapse for generative network G' is to memorize, that is, F(X,G) G) 0.
We now demonstrate how one can compute our metric and in particular m(x,; G) efficiently.
3.2 METRIC APPROXIMATION
We now turn into the actual computation of our metric. Suppose that we have generator neural network G T >> IRP that maps a latent space to data space, such as an image space. We would like measure how much of the data set is memorized and if not, how far off is the approximation in that region of the space.
We calculate m(x., G) by solving the non-convex optimization problem
min d(G(z)( (z),,, x;,
for i E (1,2,...,N) N). Since this is a non-convex optimization problem, we do not have many theoretic guarantees. This optimization is computationally expensive to run every single image. Therefore, we run it on small subset of the data to obtain an estimate riG In order to show that this estimate has low variance, we bootstrap our distance calculations calculate the standard error of mG). The variance of mmG)) is shown in the figures as errorbars and is typically low enough to justify using subset of the data. We use 100 images to calculate mG') unless otherwise stated. Moreover, for the optimization algorithm, we pick the latent variable and error [|G(z) 2 l2 that corresponds to the smallest error instead of picking the latent variable that Adam Kingma and Ba (2014) finds. This is because we can be pushed out of local minima temporarily since we are using stochastic optimization. Nevertheless, thcoretical analysis of the surface of the generator from Hand and Voroninskil (2019) suggests that such optimization can be done with great chances of reaching the global minima.
3.3 CORRELATION WITH INCEPTION AND FRECHET SCORES
In this section highlight the key differences between proposed measure and existing ones, in particular the Inception Score (IS) (Salimans et al. 2016) and its extension, the Frechet Inception distance (FID).
4

---

Under review as a conference paper at ICLR 2020
-10-2
0.3
400
Figure 1:
Depiction of our
300
0.2
300
fitness measure blue during training with yaxis on the left of digures versus the FID measure in red with yaxis on the right figures (left) DCGAN and (right) WGAN.
F(G)
FID
200
200
100
200
100
200
epoch
epoch
Figure 2:
Depiction of the closest reconstruction to the target image (left) from WGAN trained (from left to right) (i) standard, all dataset (ii) with K-means subsampling 10 (iii) with K-means subsampling K = 100 (iv) with 300 dimensional latent space dimension
current limitation of metrics for GANs is their dependence on another pretrained deep neural network model. This poses limits as to what dataset GANs can be evaluated on, as well as any internal bias induced by the different models used, On the other hand our measure does not rely on any other trained model and thus can be applied as is across different datasets. We now briefly describe those metric and then empirically observe how our metric compares FID.
The Frechet Inception distance (FID) (Heusel et al. 2017 measures how similar two data sets are. This metric is used in GAN training and evaluation because it captures, loosely speaking, how diverse the images from the generator are. However FID has two limitations: it requires trained DN and a target dataset. This means it is sensitive to new dataset applications and to the size of the dataset. We demonstrate that our memorization metric is correlated with the FID score of neural network in Figure !
We trained WGAN and DCGAN from their git repositories in order reproduce their networks. For both networks, observe that F(G) 0 meaning that do not observe any memorization in these GANs. It is specifically surprising that WGAN does not memorize because of the diversity of its generated images. This implies that even though generator can produce wide spectrum of images, it can not reproduce the training data. We provide in Fig. 2 an example of a target image and its closest reconstruction from different GAN training settings that we explore through the paper.
GENERALIZATION AND MODEL ARCHITECTURE COMPARISON
In this section we propose demonstrate some direct applications of our metric. First we demon- strate how it can be used to probe the state of generator during learning and measure how GAN fitness varies from training samples to validation samples. We then demonstrate how the measure can used to perform model comparison and in particular study the impact of residual connections and latent space dimension. Finally, we conclude by demonstrating how the optimal latent distribution of trained GAN matches the distribution imposed during training.
4.1 VALIDATION MEASURE AND EARLY STOPPING
A key problem in GAN training is knowing when stop. Either to save resources prevent overfitting, being able to probe a GAN during training and infer a stopping policy is crucial. We demonstrate in this section that our goodness of fit metric be used early stopping. In fact we demonstrate how it can be used during training to provide a meaningful signal on the state of the GAN distribution approximation. We do experiments with the DCGAN
validation metric for
5

---

Figure 3: Depiction of our measure of the GAN fitness for DCGAN (top row) and WGAN (bottom row) for the train and Under review as a conference paper at ICLR 2020
-10-2
Validation (0)
Validation (27)
0.2
Validation (42)
Validation (99)
0.1
Training Validation
100
200
100
200
epoch
epoch
0.3
Training Validation
Validation (0)
test of cluster 0 (left column) and dif- ferent cluster test set (right column).
0.3
Validation (27)
Validation (42)
F(G)
0.2
0.2
Validation (99)
0.1
0.1
100
200
0
100
200
epoch
epoch
0.2
Figure 4: Evolution the dis- tribution of the errors m((xa) during training DCGAN (top) and for WGAN (bottom)
14
914 24
49
74
99
124
249
0.6
0.4
0.2
0.0
model and WGAN in Fig. 3 We obtain the following conclusions. The DCGAN favors a minority of points making the overall FGG) measure first decrease and then increase when this specialization occurs. In fact, the DCGAN will favor learning exactly some images and less good on average. On the other hand WGAN is much more robust such degeneracy and thus has performances stable even if training for a long period of time. As such, in the DCGAN case our metric can be used to do early stopping and prevent decrease overall performances, the WGAN setting allows to save resources by stopping when gains appear even though the performance do not get worse with time. We also provide Fig. 4 the evolution the histograms the errors m(xx,) for both settings highlighting how for the DCGAN, the main reason for increase of the approximation error lies in outliers being more and more apart from Imag(G).
4.2 MODEL COMPARISON
In this section we propose to study the impact of the architecture onto the goodness fit of the generator. To do so experiment adding residual connection onto DCGAN. In Figure two WGAN networks were trained on CIFARIO (Krizhevsky et al. 2009) and the LSUN (Yu et al, 2015) bedrooms dataset.

---

Under review as a conference paper at ICLR 2020
0.4
no res
n1 res res
res
0.2
0.3
Figure 5: WGAN experiment on CIFARIO (left) and LSUN (right) for the same network with and without residual con- nections
0.2
0.15
0.1
2,000
4,000
10
20
epoch
epoch
Hissogram of the compenents of
Hissogra of the components of
Figure 6: The initialization and the final z". For illus- trative purposed, the wide dis- tributions the left shown with smaller standard devia- tions than in practice.
-G
[e]]
[<"]
4.3 LATENT SPACE DISTRIBUTION ANALYSIS
Recall that we solve the optiiization problem x¹ = arg min [[((z) æ|12 in order to try and generate æ. A natural question what the distribution of x In this setting we have GAN with latent space distribution z No0, I). In fact, while the optimization algorithm can be initialized with normal independent and identically distributed random variables, might be possible to have less likely that would allow better reconstruction. However we demonstrate in figure 6 that the actual distribution GAN is trained with is also the one that will minimize the overall goodness of fit metric. That is, once trained with specific distribution, changing this distribution. This brings an interesting observation that the limit in the reconstruction and span of the generator comes form the model and weights themselves rather the distribution.
not increase performance just by
REDUCING A GAN'S APPROXIMATION ERROR
In this section we study the two proposed strategies to decrease the approximation error of GANs. First in Sec. [.1 we demonstrate how the dimension of the latent space be increased. Then in Sec. 5.2 we demonstrate how reducing the size of the dataset allows to better fit the data and then build upon this finding experiment with mixture of GANs in Sec. 5.3
5.1 INCREASING LATENT DIMENSION
We now turn into the latent space dimension itself. It has been 100 across very large number of networks and dataset. However, recall that Imag(G) depends on the latent space itself and thus incidentally on its dimension. We thus experiment with two different setting, the standard and the expanded e 300 and this on CIFAR10 and LSUN dataset. We report the results in Fig. 7 where one can observe how increasing the dimension naturally allows to reduce the approximation error of the generator.
100
5.2 DATASET SUBSAMPLING HELPS
We demonstrated above that current GANs not span the training set and in general seem limited by their generative network. We propose this section to study how dataset subsampling affects measure. As the generator capacity is limited reduce the complexity of the data and observe

---

Under review as a conference paper at ICLR 2020
no res 100
100
0.4
no res 300
0.2
res 300
res 100
100
Figure 7: WGAN
0.3
F(G)
res 300
0.15
res 300
experiment on CIFARIO (left)
0.2
and LSUN (right) for the same network with and without residual connections.
0.1
0.1
2,000
4,000
10
20
epoch
epoch
30
K100
normal K10 mixture
40
K1OR10 R100
20
20
0
0
0.2
0.4
0.6
0.2
0.4 0.6
[m(G)]
(m(G))
Figure 8: (left)): Histograms of m(æ,, G7) for different subsampling methods. We notice that K100 method is the best among the three in terms of goodness of fit. We perform random subsampling (R100), kmeans clustering (K100), and mixture with 10 clusters and then subsample randomly by 10 in each of those clusters. (right) Histograms of m(x;; G) for WGAN our mixture WGANs
two (K10R10).
KIORIO case, we first use kmeans clustering
the new measure of F(G) after training. We obtain the key following conclusions. As we sub- sample our data and our parameters to data ratio increases, our goodness of fit measure decreases. Clustering images that are close in the input space allows for better goodness of fit measure, even though we are in very high dimensional setting. That is, the number of points matter but also the similarity between those points. This a key insight that might play a role explaining conditional GANs (Mirza and Osindero 2014) that act subsampling by forcing each class to have its own set of parameters.
5.3 MIXTURE OF GANS
In this section leverage the finding from the previous section and exploit K-means based parti- tioning of LSUN with K 10 to subsample the dataset into 10 non overlapping subsets. We then propose to train independent WGAN each of the subset then form the overall generative network being a combination of the per subset WGANS. The sampling strategy thus becomes (i) randomly sample one of the subset (ii) randomly generate an image from the WGAN of the picked subset. This two step sampling can be assimilated to mixture model such as Gaussian Mixture Model. We obtain the following approximation error when performing such training in Fig.
6 CONCLUSION
We defined a goodness of measure F(G) for generative networks. We used FGG) to show that DCGAN and WGAN fail to memorize the training data even though they can generate compelling images. We provide means to get a better goodness of fit measure by subsampling data randomly or with K-means clustering.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv: 077775,. 2017.
Wasserstein
gan. arXiv preprint
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning and the bias-variance trade-off. arXiv preprint arXiv: 18/2. 1 11188, 2018.
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. arXiv preprint 1rXiv:1707.057700 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pages 2672-2680, 2014.
Paul Hand and Vladislav Voroninski. Global guarantees for enforcing deep generative priors by empirical risk. IEEE Transactions on Information Theory, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. Multi-generato, generative adversarial nets. arXiv preprint 17Xx:17088002556 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 0rXiv:1412.6990 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Ke Li and Jitendra Malik. Implicit maximum likelihood estimation. arXiv: 180900977 2018
arXiv preprint
Zachary C Lipton and Subarna Tripathi. Precise recovery of latent vectors from generative adver- sarial networks. arXiv preprint arXiv: 1702.0442,2 2017.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1788 2014.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06433 2015.
Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Memorization in overparam- eterized autoencoders. 2019.
Amit Rege and Claire Monteleoni. Evaluating the distribution learning capabilities of gans. 2019.
Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, 2016.
pages
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint / 536,03355,, 2015.


Reviews:
[{"review_id": "SygOTfanYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper defines a goodness of fit measure F for generative networks, that reflects how well a model can generate the training data. F allows to detect mode collapse: as long as it is strictly positive, mode collapse is observed as parts of the training data have not been memorized. It aims at providing an alternative to the Fréchet Inception Distance and the Inception Score that rely on pretrained neural networks (whereas this new measure does not). It also provides insight into the DCGAN and WGAN networks in that regard, observing for instance that data subsampling helps decrease F, which motivates the use of a mixture of GANs.\n\nThis paper brings an interesting contribution to the evaluation of generative networks. However:\n\n1.\tThe use of the square distance in the image space is not obvious and not justified.\n2.\tComputation of this metric is not straightforward: there is no theoretical guarantee and it is computationally expensive.\n3.\tThe theoretical properties of this measure and its robustness are not investigated.\n4.\tTypos are obscuring the reading of the paper.\n\n- Post rebuttal: I have read the authors' response and am maintaining my weak reject rating.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlxTZnitB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposed a new goodness of fit measure for generative network evaluations, which is based on how well the network can generate the training data. The measure is zero if the network could perfectly recover the training data, and would represent how far it is from generating the training set in the average manner of the total least square sense, where the one-to-one mapping between the generated data and the training sample is constructed through latent space optimization. Using the proposed measure, the authors showed an interesting trend present in the DCGAN training and the impact of the residual connection. The authors might want to add some discussion in Section 4.2 regarding why the residual connection is detrimental for covering the support.  Increasing the model complexity through larger latent space dimension and learning mixtures is proposed as solutions to improve the measure as well.\n\nWith all the interesting results presented, I still have the concerns about the sensitivity of the proposed measure:\n- It is an average over the training data or the selected sample. Above Section 4, the authors argued that \"\\hat{F}(G) > 0 meaning that we do not observe any memorization\". This seems overly assertive. Since the measure is an average over the training data, it has difficulty to differentiate between one network which has almost zero value for part of the training data but large values for the rest, and another network with roughly the same \\hat{F}(G) value but small values for all training data. The variance could help, but can not resolve this issue. This would be more important when the training data contains noise or outliers.\n- It only concerns the generation of the training data, but not the sampled data from the network (at least not directly). Therefore it has no direct control of the fidelity of the generated samples.\n- As shown by the authors, the proposed measure can be considered as the approximation of the true probability support not covered by the generative models, which also defines a necessary condition to avoid mode collapse. But what about the other part? It would have difficulty comparing two models with the same support but different high-density areas. Indeed, there are existing works which consider both the precision and recall of the generative models [1, 2, 3], and directly work with the generated samples instead of the training data. These should be discussed and compared with, not just the FID scores which have already been shown to have issues [3]. \n\nSome notations:\n- In the last equation on Page 2,  should it be L_{G} instead of L_{D}?\n- In the first equation on Page 3, should the denominator be N_{B} instead of N_{N}?\n- \"Optimality\" in terms of generative models may depend on the downstream tasks. I do not think there exists a universal definition of \"optimality\" for generative models.\n\n[1] M.S.M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. NeurIPS 2018.\n[2] L. Simon, R. Webster, and J. Rabin. Revisiting precision and recall definition for generative model evaluation. ICML 2019.\n[3] T. Kynkaanniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. Arxiv:1904.06991.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bkx7i9i1KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new goodness of fit measure for generative models, and uses it to get insight into GAN's. While this is an important topic and a novel approach, I do not think the paper delivers on what it promises.\n\nI think this paper should be rejected. First, while it claims to be a general method for generative models it, it is limited to only  GANs and even for GANs it is limited. Second, most of the observations are nice but trivial, e.g. larger latent space leads to larger image.\n\nDetailed remarks:\n- The main point that the training set points x must have p(x)>0 under the model is naturally satisfied for almost all models except GANs such as VAEs, autoregressive model and flow models with standard implementations as the support is the whole space. This is in contrast to the claim in the paper that \"its applications can be extended to other generative networks such as Variational Autoencoders.\".\n- Even for GANs as this measure only looks at the support and not the distribution it is not clear if this measure does more then evaluate mode collapse. While this is an important task, it falls short of the promises the authors claim.\n- The authors claim that \"We demonstrate that our measure being minimized is a necessary and sufficient condition to detect mode collapse.\" but only show that it is necessary.\n- Proposition 1 is a trivial statement.\n- The authors claim that \" mode collapse happens if P(x) > 0 but minz ||G(z) − x|| > 0\". This is a main point by the authors, but it ignores the probability and only looks at the support. It has been shown that mode collapse happens even in 2d distributions, e.g. veegan paper, where it is easy to get the support to be the whole distribution.\n- The results in sec. 5 are quiet obvious, with a larger latent space you can naturally get a larger support, same as with a mixture model.\n\n\nIn general the method only looks at the support, ignoring the distribution over the support and is therefore very limited in evaluating generative models.  \n\n\nminor details:\n- In eq. 3 the integration should be w.r.t dP(x) for it to be monte-carlo approximated as it is in eq. 4.\n- Not 100% I understand what the authors try to say here - \"we pick the latent variable z and error ||G(z) − x||2 that corresponds to the smallest error instead of picking the latent variable that Adam Kingma and Ba (2014) finds.\"", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to measure the distance of the generator manifold to the training data. The proposed approach bears significant similarity to past studies that also sought to analyze the behavior of generative models that define a low-dimensional manifold (e.g. Webster 2019, and in particular, Xiang 2017). I recommend that the authors perform a broader literature search to better contextualize the claims and experiments put forth in the paper.

The proposed method also suffers from some limitations that are not made clear in the paper. First, the measure depends only on the support of the generator, but not the density. For models that have support everywhere (exact likelihood models tend to have this property by construction), the measure is no longer meaningful. Even for VAEs, the measure is only easily applicable if the decoder is non-autoregressive so that the procedure can be applied only to the mean decoding. 

In this current state, I do not recommend the paper for submission.

Xiang (2017). On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
Webster (2019). Detecting Overfitting of Deep Generative Networks via Latent Recovery

Decision: Reject