Title: IMPACT OF THE LATENT SPACE ON THE ABILITY OF GANS TO FIT THE DISTRIBUTION Anonymous

Abstract: The goal of generative models is to model the underlying data distribution of a sample based dataset. Our intuition is that an accurate model should in principle also include the sample based dataset as part of its induced probability distribution.
To investigate this, we look at fully trained generative models using the Generative Adversarial Networks (GAN) framework and analyze the resulting generator on its ability to memorize the dataset. Further, we show that the size of the initial latent space is paramount to allow for an accurate reconstruction of the training data. This gives us a link to compression theory, where Autoencoders (AE) are used to lower bound the reconstruction capabilities of our generative model. Here, we observe similar results to the perception-distortion tradeoff (Blau & Michaeli (2018) ). Given a small latent space, the AE produces low quality and the GAN produces high quality outputs from a perceptual viewpoint. In contrast, the distortion error is smaller for the AE. By increasing the dimensionality of the latent space the distortion decreases for both models, but the perceptual quality only increases for the AE.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) were introduced by Goodfellow et al. (2014) for the purpose of generative modelling. Since then this framework has been successfully applied to works in style transfer by Karras et al. (2018) , superresolution by Shocher et al. (2018) and semi-supervised learning by Salimans et al. (2016) , but what GANs actually learn is still poorly understood as has been noted by Webster et al. (2019) . Recently, GANs have been used to solve inverse problems, where it was tried to use the generated manifold to solve an auxiliary task like image completion (Webster et al. (2019) ), MRI reconstruction (Narnhofer et al. (2019) ) or anomaly detection (Shocher et al. (2018) ). For those applications, it is necessary to know if the generator NN actually describes the distribution well. Related works have shown that faithfully reconstructing the images from a generator network is not trivial (Webster et al. (2019) ; Shmelkov et al. (2018) ).

The original convergence proof by Goodfellow et al. (2014) assumes that the generator and discriminator Neural Networks (NNs) have infinite capacity and they showed that the discriminator network models the Jensen-Shannon divergence between the probability distribution induced by the generator and the real data distribution. Others have adapted this paradigm and devised loss functions which have been shown to converge to other divergences or distances on the underlying probability distributions (Arjovsky et al. (2017) ; Nowozin et al. (2016) ; Mao et al. (2017) ). Regularization techniques like Gradient Penalty (Gulrajani et al. (2017) ) and Spectral Norm (Miyato et al. (2018) ) did improve the stability of the training process (Kurach et al. (2018) ) but it is still unclear how well this NNs actually approximate such distances even for trivial problems (Pinetz et al. (2018) ). Additionally, it is not at all clear how the generated distribution or the actual target distribution look like. Arora & Zhang (2017) used the birthday paradox to empirically gauge the size of the support of the generated distribution.

GANs are used to transform a well understood low dimensional distribution (in practice either gaussian or uniform) to a high dimensional unknown distribution (Goodfellow et al. (2014) ) by playing a min-max game between two NNs. This paper is based around the intuition, that an estimated probability distribution from a dataset X has high precision if a high percentage of the actual data samples are included in the estimated distribution. To have a sense of what an adequate capacity for a generator network is, we use AE to reconstruct the dataset first. This work relies on the assumption that it is easier to reconstruct the data samples alone, than to reconstruct the entire manifold and Section 5 shows empirical evidence for this. Based on our intuition the manifold consists of the data samples and imposes additional structure between the data samples. In contrast by just reproducing the data samples, no such additional restrictions are given, making the problem strictly simpler. AEs can be trained rapidly and have been researched in detail for a long time (Rumelhart et al. (1985) ). In contrast, trying to do a hyperparameter search on the GAN networks themselves gives rise to all kinds of problems, like instabilities in the training process, random failures and dependence on random seeds for their performance (Lucic et al. (2018) ).

Hence, our contributions are as follows:

• An investigation of the impact of the dimensionality of the latent space on the generated manifold. We showcase that the fit of the data depends heavily on the latent space. We also show similar results thereby to the perception-distortion tradeoff (Blau & Michaeli (2018) ), where with a small dimension for the latent space, the GAN optimizes for perception and the AE optimizes for distortion. • Relating the GAN problem to a compression task and furthermore using compression tools via deep learning to produce a lower bound for a dataset dependent suitable dimensionality of the latent space. • An investigation of the generated manifold and the limitations thereof to produce shifted or noisy images and how this relates to the size of the latent space and overfitting of the generative model.

The remainder of this paper is organized as follows. Section 2 shows the related work. Then in Section 3 we revisit the theory behind pseudo inverting NNs and we explain our methodology in Section 4. In Section 5 the results are shown. Section 6 draws conclusions based on those results.

2. RELATED WORK:

GANs have been invented by Goodfellow et al. (2014) for the task of image synthesis and semisupervised learning. The initial formulation has been shown to suffer from mode collapse and vanishing gradient problems and multiple other loss functions have been proposed (Nowozin et al. (2016) ; Mao et al. (2017) ; Arjovsky et al. (2017) . We choose the Wasserstein GAN, due to it using the L2-distance as its transport cost function to judge the quality of the generated manifold, which is similar to the Mean Squared Error (MSE) used in the classical AE (Rumelhart et al. (1985) ).

To satisfy the Lipschitz constraint we use Gradient Penalty Gulrajani et al. (2017) , due to its good performance in review works (Lucic et al. (2018) ; Kurach et al. (2018) ).

For evaluation purposes we need a quality metrics for GAN and the most common ones rely on the usage of classification models, which are trained on ImageNet (Deng et al. (2009) ). Two commonly used metrics are the Inception Score (Salimans et al. (2016) ) and the Fréchet Inception Distance (FID) (Heusel et al. (2017) ). To combat the bias of having trained on ImageNet, Shmelkov et al. (2018) proposed to train a new classifier to evaluate the GAN model, but this is time consuming and prone to errors by not having a standardized metric. We opted for the FID score, due to its prevalence in related works (Lucic et al. (2018) ; Kurach et al. (2018) ).

AEs have been studied extensively in the past and initially have been proposed by Rumelhart et al. (1985) . They have found usage in compression literature (Cheng et al. (2018a; b) ; Li et al. (2019) ).

Their first usage in a GAN framework has been shown in the Energy based GAN by Zhao et al. (2016) . Since then they have been used as preprocessing to reduce the dimensionality of the data (Guo et al. (2019) ), as regularizers (Che et al. ( 2016)) or as generative models themselves (Webster et al. (2019) ).

The inversion of generator networks has been initially used by Metz et al. (2016) and there have been multiple works on how to improve their method (Creswell & Bharath (2018); Lipton & Tripathi (2017) ) by adding heuristics like stochastic clipping on top of that. We opted against stochastic clipping, because we already project back to the likely set of candidates. Stochastic clipping would add even more randomness to the results. Generalization of GANs through the detection of overfitting has been proposed recently by Webster et al. (2019) . Therein the authors are mostly concerned with detecting memorisation of the training set in the manifold and avoid said memorisation to generalize. In contrast we make the assumption that an estimated probability distribution based on samples should include those samples and overfitting is detected based on a hold out set as well as the ability to reconstruct that hold out set as well as the training set. Additionally, we show that the ability of the GAN to memorize depends mostly on the dimensionality of the latent space and the capacity of the generator network and can be estimated using compression techniques.

3. PRELIMINARIES: WHEN ARE NNS INVERTIBLE?:

In this work, we want to show the fit of the distribution of trained GAN newtworks using inversions of such networks. This motivates the question of "When are NNs invertible?". Due to the nonconvex and non-bijective nature of NNs using ReLU activations invertibility is not given in the general case. The function induced by the NN is in general neither injective nor surjective and it is trivial to construct such cases Arora et al. (2015) .

However, if the weights are gaussian distributed, proofs exist, which guarantee that the NN is invertible with a high probability. For this, we revisit the inversion theory of NNs largely pioneered by Arora et al. (2015) . They empirically showed that the weights of trained NNs actually exhibit random like properties, enabling the usage of Random Matrix Theory (RMT) to investigate such networks. Theorem 1 in (Ma et al. (2018) ) proves that we can invert 2-layer NNs with high probability. In their work they empirically showed that this also holds for multi layer networks. In Figure 1 we show that the same holds true for trained GAN networks. Note, that this property holds across datasets and latent spaces used to train those networks. Glorot initialization (Glorot & Bengio (2010) ) was used to initialize these networks. Therefore, the standard deviation used for the initialization process differs for each layer. However, as is demonstrated in Fig. 1 , all layers converged to a similar gaussian distribution.

Another indicator, that the optimization process is likely to succeed is given by a trick proposed by Li et al. (2018) for classification networks. It allows us to visualize the loss surface of the optimization problem for one sample at a time. For a particular data point z * , we chose two orthogonal vectors α, β and plot the following 2D function:

EQUATION

We chose α = z * -z 0 to model the optimization path towards the original data point. For this experiment we do not need to run an optimization, because the point is to show that it is possible 

4. ESTIMATING A GOOD GAN MODEL FOR A DATASET:

Currently, designing new GAN networks works by running a huge hyperparameter search on a plausible set of combinations (Lucic et al. (2018) ; Kurach et al. (2018) ). Lucic et al. (2018) noted that the progress in GANs has actually been quite minimal and vastly overstated, and depends on the hyperparameters more than on the improvements in the methodology, while Kurach et al. (2018) did notice that non-saturating loss in combiniation with spectral norm produced consistently good results, but never the best ones.

4.1. LOWER BOUND ON THE GENERATOR NETWORK:

The goal of the GAN framework is to encapsulate the high dimensional image distribution in a lower dimensional noise distribution, whereas a generator network maps the lower dimensional distribution to the former high dimensional image distribution (Goodfellow et al. (2014) ). This implies that we compress the larger distribution onto the simple lower dimensional distribution and use the generator network as an approximation for the decompression. We rely on the assumption, that less capacity is needed to reconstruct the training set, than to reconstruct the entire manifold. Therefore, to gain some insight into the necessary capacity of the generator network, the usage of an AE is proposed to gauge the number of parameters and the size of the latent space to enable suitable approximation of the training set. In theory, a latent dimension of 1 would be sufficient to reconstruct the entire dataset, by mapping a curve to the datapoints. A simple smooth function that does this for a dataset {x 1 , .., x n } = X ∈ R n×d and {p 1 , ..., p n } = P ∈ R n×1 looks like this:

EQUATION

This function will output the corresponding image x i for every p i . In practice, this behavior is not observed. AEs with small latent spaces tend to output blurry approximations of the training images as is shown in Section 5. Once a suitable AE for a dataset is found, the decoder part of the AE is used as the generative model of choice. The AE is trained using the L2-loss, which also is the transport cost function used in WGANs (Arjovsky et al. (2017) ) to gauge the quality of the generated manifold. Therefore, the objective function of the AE using a decoder d and an encoder e is defined as follows:

L = min d,e

||d(e(x)) -x|| 2 (3)

We do not opt for any regularization, because we want to fit the data as closely as possible to determine if the generator capacity is adequate for reconstructing the dataset. In this work we show empirically in Sec. 5 that if the AE already fails to produce crisp results, then also the GAN algorithm fails at having the dataset included in the generated manifold. One thing to note is that, while the encoder part is used as our discriminator, it has a completely different objectives and it is unclear, if a better encoder is also a better discriminator.

4.2. DETERMINING THE FIT OF THE DATA IN GANS:

The key part of this work is to analyse the ability of the GAN to memorise the training dataset, following the assumption, that a probablity distribution based on a sample dataset has to include that dataset. To our best knowledge there is no analytical way to invert generative NNs. Therefore this ill posed problem is stated as an inverse optimzation problem for a function g:

z * = arg min z ||g(z) -x|| 2 2 (4)

This is solved using a standard first order gradient method. However, as is shown in Section 5 the resulting latent vectors are unlikely to stem from the targeted noise distribution, if there are no constraints added. Therefore, the search space for z is restricted to the space of plausible latent vectors. Plausibility in this context means that the probability of the latent vector originating from this space is larger than 99.7% (3-sigma-rule). In the standard multivariate gaussian distribution the norm of the vector z corresponds inversely to the likelihood of z ∈ N (0, I). Therefore we restrict the norm to be ||z|| ≤ E[||z||] + 3 V AR [||z||] . For the multivariate gaussian distribution this is easy to calculate and the final objective function is given as follows:

EQUATION

This is solved using a projected ADAM optimizer (Kingma & Ba (2014)).

5. EXPERIMENTS:

Our experiments are done on medium sized datasets (CIFAR-10 and CelebA (64 × 64 resolution)) and the proof of concept on MNIST is relegated to the supplementary material. The NN adapted for this task is the standard DCGAN from Salimans et al. (2018) . Convolutional layers and resize operations are used instead of transposed convolutions, to avoid checkerboard artifacts (Odena et al. (2016) ). The full hyperparameter setting is shown in the supplementary material. The visual quality measured by the FID (Heusel et al. (2017) ) of the resulting samples is on par with the survey work by Lucic et al. (2018) . This work is not aimed at improving the visual quality of the samples generated by GAN algorithms. For CIFAR-10 the test set is used as our validation set and for CelebA we randomly split off 20% of the dataset to be used as validation set.

5.1. RELATIONSHIP BETWEEN AE AND GAN RECONSTRUCTION LOSS:

In principle, getting from an AE to a GAN is just a rearrangement of the NNs. For the GAN training, the default hyperparameter are used as in Gulrajani et al. (2017) . The AE NNs are optimized using the ADAM(Kingma & Ba ( 2014)) optimizer with default hyperparameters for all the datasets, always on a single GPU. A comparison of the resulting reconstruction errors is shown in Fig. 3 . Therein the different AE and GAN models only differ in their latent space sizes. Increasing the latent space dimension also increases the number of parameters. However, for the sake of simplicity this was neglected. The AE network acts as a lower bound for the GAN algorithm, therefore validating our intuition that the AE complexity lower-bounds the GAN complexity.

5.2. VISUAL QUALITY OF GANS VS AE GENERATED IMAGES:

The visual quality of the generated images is evaluated using the FID (Heusel et al. (2017) ) versus the reconstructed images by an AE for the same latent space sizes. The experimental results are shown in Fig. 4 for the CIFAR and CelebA datasets and demonstrate, that the visual quality of the GAN generated images barely changes with different latent dimensionality. In contrast, the AE reconstructions gain quality and sharpness as the latent space increases. This reflects the output of those models shown in Fig. 5 , where the resulting images are clearly an approximation of the actual training images. On the other hand, the GAN reconstructions have similar visual quality independent of the actual latent space. However, the resulting images are only remotely similar to the actual image for insufficiently large latent spaces. This phenomenon has been observed in other works, where a similar looking person to an actual training image is found by the inversion procedure (Webster et al. (2019) ). However, we show in Fig. 5 , that this phenomenon depends mostly on the size of the latent space. 2017)) curve for different latent spaces (lower is better). Increasing or decreasing the latent space does not result in a noticeable difference in the quality metric for the WGAN-GP, but it does for the AE.

5.3. WHAT IS INCLUDED IN THE GENERATED MANIFOLD?:

While it is possible to reconstruct the dataset accurately using a large enough latent space, in this section we look at what else those models can reconstruct. In these experiments, our baseline model Figure 6 : Impact of translation on the reconstruction ability of GAN networks. Notice, that by increasing latent space dimension, the generator network is able to reconstruct translations with a similar precision as the original images.

The second experiment is on reconstructing the CIFAR-10 dataset using the CelebA trained model.

To provide the same resolution, the CIFAR-10 images were upscaled to 64 × 64 using bilinear interpolation. If the latent space is large enough, we can reconstruct the images well, as shown in Fig. 7 . The reconstruction of the CelebA images is always worse as measured by the MSE (Fig. 7 ).

The final experiment is on reconstructing noise images. For this a dataset of uniform noise images with size 64 × 64 is used. The results are shown in Fig. 8 . As is suggested by the results, it is in fact not possible to reconstruct everything. One thing to note though is that with lower latent spaces some face structure is visible in the reconstructed images, showing that those models have a strong prior on the face structure. We noticed and the corresponding experiment is shown in the supplementary material, that the norm of the latent vector is smaller than of random vectors. Our hypothesis is that those images are the base face features, which are refined with larger latent vectors, but this guess needs further validation work. 

6. CONCLUSIONS:

In this work, we show that by reducing the problem to a compression task, we can give a lower bound on the required capacity and latent space dimensionality of the generator network for the distribution estimation task. Relating these two different algorithms to each other, the literature surrounding AEs for invertability and dimensionality reduction, as well as the corresponding theoretical advancements are used. While in this work the encoder and the discriminator NNs use the same architecture, we have not discovered any relation between them. Still, the same architecture works well empirically for both task.

Using our framework we show various properties of generator networks. The perceptual image quality appears to be independent of the actual size of the latent space, which is in contrast to AE, where the visual quality improves if the dimensionality of the latent space is increased. However, the ability to reconstruct the training set correctly does depend on the initial latent space. Also the ability of the generator to reconstruct deviations from the original dataset, like a validation set or shifted images depends just as much on the initial latent space. However, the same cannot be said for reconstructing arbitrary noise images. Here the reconstruction ability is independent of the initial latent space unless it is chosen very large, suggesting that the generator has learned realistic natural image features. Here for smaller latent spaces we can still observe face like features. Our hypothesis is that the implicit bias induced by the generator architecture lends itself to generating natural images and GANs are skilled at that by learning primitives which can be combined to construct arbitrary images. In future works we want to use our setup to search towards better and more reliable generators for images. The initial experiments on MNIST validating the legitimacy of the approach are shown in Fig. 9 . Instead of using a visual quality metric to determine if the resulting images are similar enough, we used the reconstructed training images of the GAN for training a LeNet network LeCun et al. (1998) to determine if the images are descriptive. The manifold of MNIST has been faithfully learned before by Shmelkov et al. (2018) . In contrast to their work, we show that this only works given a large enough latent space and that the reconstruction error of the AE and the WGAN-GP converge for larger latent spaces. 

:



Figures and Tables:

Figure1: Histogram of the values in the weight matrix and the convolutional layers of a trained WGAN-GP with a latent space specific dimensionality for CIFAR-10 and CelebA datasets. Notice, that the weights share the same characteristics as if they were drawn from a normal distribution.

Figure 2: Loss surface around a data point of the generator of a WGAN-GP trained on CelebA.There is a clear optimization path from z 0 = 0 to a point z * even in this simplified plot, empirically validating the claims ofLipton & Tripathi (2017), that an accurate reconstruction of the generator manifold is possible works using first order methods.

Figure3: Impact of the latent space on the AE and GAN reconstructions loss on CIFAR-10 and CelebA. The AE acts as a lower bound for the GAN reconstruction loss and the reconstruction loss decreases for both the AE and the GAN generator with increasing latent space dimensions.

Figure 4: FID(Heusel et al. (2017)) curve for different latent spaces (lower is better). Increasing or decreasing the latent space does not result in a noticeable difference in the quality metric for the WGAN-GP, but it does for the AE.

Figure5: The first columns show the original images. As we increase the latent space of the autoencoder the image quality increasing. The GAN reconstructions are of high perceptual quality even for low dimensional spaces, but correspond to other people as has been observed byWebster et al. (2019).

Figure 7: Left & Right: Reconstruction of CIFAR images in a WGAN-GP model trained on CelebA images.As the latent dimensionality is increased, the reconstruction is improved for both CIFAR-10 and CelebA. Right: With a small latent space, the face image features are retained, while with a large latent space, the CIFAR-10 images are faithfully reconstructed.

Figure 9: Mnist GAN and AE reconstructions in (a). Visual quality metrics are not really applicable on MNIST data so in (b) the reconstructed GAN images are used to train a LeNet to classify on MNIST and is compared to training on the true training set.

Architecture of the Discriminator and Generator networks. Initialization is done usingGlorot & Bengio (2010) PROOF OF CONCEPT ON MNIST

Table content: <table/>

Peer review (iclr2020):
Impact of the latent space on the ability of GANs to fit the distribution This paper purports to study the behavior of latent variable generative models by examining how the dimensionality of the latent space affects the ability of said models to reconstruct samples from the dataset. The authors perform experiments with deterministic autoencoders and WGAN-GP on CIFAR, CelebA, and random noise, and measure MSE and FID as a function of the dimensionality of z. My take: This paper does not offer any especially intriguing insights, and many of the conclusions the authors draw are, in my opinion, not supported by their experiments. The paper is confusingly written and hard to follow—throughout my read I struggled to determine what the authors meant, and it was not clear to me what this paper is supposed to contribute. The potential impact of this paper is very low, and I argue strongly in favor of rejection. Notes: My most critical complaint is the central experiment set of the paper: measuring MSE and FID as a function of Dim-Z for two models. First of all, the authors assume that the reduction in MSE as a function of dim Z is indicative of increased memorization in the GAN models. I disagree that this is the case; since the GAN-based reconstruction is done via optimization it is unsurprising that increasing dim Z increases the reconstruction quality, as you are literally giving the model more degrees of freedom with which to fit the individual samples. This is glaringly evident in Figure 8, where increasing dim Z renders the model better able to reconstruct a sample of pure noise, which is almost certainly not in its normal output distribution, (or if it is, is in there with staggeringly low probability). The fact that the higher dim-z models are better able to reconstruct the noise supports the notion that it is merely the number of degrees of freedom that matter in these experiments, rather than what the model actually learns. Second, it is important to note that FID can be easily gamed by memorization, and for an autoencoder (which has direct access to samples) with an increasingly large bottleneck it is unsurprising that increasing dim-Z tends to decrease the FID, and equally unsurprising that increasing the dim-Z for the GAN does not tend to improve results, since this does not really allow the model increased memorization capacity (not to mention the relationship between performance and dim-Z has been explored before in GAN papers). Third, the organization of the experimental section makes it very difficult to infer what the authors are trying to conclude from these experiments. The noise experiment is presented, but no insights or conclusions are drawn, other than (a) noting that the model has a harder time reconstructing the noise than training samples and (b) that lower dim models have a harder time reconstructing the noise, both of which are just restatements of the information presented in the figure rather than an actual insight or conclusion. -I’m not really sure what the experiment in section 3 is supposed to show. This experiment is poorly described and lacking details. First of all, what is the loss function used there? Is this the output of the discriminator or the MSE between the output and a target sample? How is z* found and what does it represent—is it just a randomly sampled latent, or is it the latent that corresponds to the z-value which minimizes some MSE loss for a target sample? If it’s the latter, why is this notation not introduced until section 4? If it’s a latent, why are you calling it a data point? Why are there no axes and no scales on these plots? How is it clear that there is an optimization path from z0 to z*; is that supposed to be inferred from z0 having a higher value than z* or appearing to be directly uphill from z*, because it’s not clear to me that that is the case in Figure 2a. In general I did not find this experiment to support the conclusions the authors draw. -Figure 4: It is important to note that FID can be trivially gamed by memorizing the dataset, and an autoencoder is much more well-suited to memorizing the dataset as it has direct access to samples (whereas a GAN must get them through the filter of the discriminator). Authors should test interpolation or hold-out likelihood for the autoencoder, these models are not directly comparable in this manner. -The presentation of this paper is, in general, all over the place. The authors should focus on writing such that each point follows the next, building progressively towards their results and insights, and making it easy for a reader to follow their train of thought. “In this work, we show that by reducing the problem to a compression task, we can give a lower bound on the required capacity and latent space dimensionality of the generator network for the distribution estimation task.” At what point is this lower bound (either in terms of model capacity or latent space dimensionality) specified in the paper? Is Figure 3 supposed to be this lower bound, because to me it only indicates that the autoencoder tends to have a lower MSE, not that it conclusively lower bounds the memorization capacity of the GAN. Wouldn*t a method like GLO which directly optimizes for memorization be a better lower bound for this, anyhow? “We rely on the assumption, that less capacity is needed to reconstruct the training set, than to reconstruct the entire distribution” What does this phrase mean? Are the authors referring to the entire distribution of natural images, of which the training set is assumed to be a subset? Or do they mean the output distribution of the generator? This was not clear to me. Minor: -“ style transfer by Karras et al. (2018),”, and “anomaly detection (Shocher 2018).” StyleGAN is not a style transfer paper, and InGAN is not about anomaly detection. Please do not incorrectly summarize papers. -“Trained GAN newtworks” While amusing, this is a typo. Please thoroughly read your paper and correct all typos and grammatical mistakes, like “combiniation.” -“…that an accurate reconstruction of the generator manifold is possible works using first order methods” The word “works” seems to be out of place here. Again, please thoroughly proofread your paper. -The legend in Figure 2 has a white background, making the white x corresponding to z0 invisible. Please fix this, and add appropriate axes to this plot. -Figure 7 and 8 may in fact have error bars, but they are not described (are they 1 std or another interval?) or referenced, and in Figure 8 (if these are error bars) they are nearly invisible.