Title: IMPACT OF THE LATENT SPACE ON THE ABILITY OF GANS TO FIT THE DISTRIBUTION Anonymous

Abstract: The goal of generative models is to model the underlying data distribution of a sample based dataset. Our intuition is that an accurate model should in principle also include the sample based dataset as part of its induced probability distribution.
To investigate this, we look at fully trained generative models using the Generative Adversarial Networks (GAN) framework and analyze the resulting generator on its ability to memorize the dataset. Further, we show that the size of the initial latent space is paramount to allow for an accurate reconstruction of the training data. This gives us a link to compression theory, where Autoencoders (AE) are used to lower bound the reconstruction capabilities of our generative model. Here, we observe similar results to the perception-distortion tradeoff (Blau & Michaeli (2018) ). Given a small latent space, the AE produces low quality and the GAN produces high quality outputs from a perceptual viewpoint. In contrast, the distortion error is smaller for the AE. By increasing the dimensionality of the latent space the distortion decreases for both models, but the perceptual quality only increases for the AE.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) were introduced by Goodfellow et al. (2014) for the purpose of generative modelling. Since then this framework has been successfully applied to works in style transfer by Karras et al. (2018) , superresolution by Shocher et al. (2018) and semi-supervised learning by Salimans et al. (2016) , but what GANs actually learn is still poorly understood as has been noted by Webster et al. (2019) . Recently, GANs have been used to solve inverse problems, where it was tried to use the generated manifold to solve an auxiliary task like image completion (Webster et al. (2019) ), MRI reconstruction (Narnhofer et al. (2019) ) or anomaly detection (Shocher et al. (2018) ). For those applications, it is necessary to know if the generator NN actually describes the distribution well. Related works have shown that faithfully reconstructing the images from a generator network is not trivial (Webster et al. (2019) ; Shmelkov et al. (2018) ).

The original convergence proof by Goodfellow et al. (2014) assumes that the generator and discriminator Neural Networks (NNs) have infinite capacity and they showed that the discriminator network models the Jensen-Shannon divergence between the probability distribution induced by the generator and the real data distribution. Others have adapted this paradigm and devised loss functions which have been shown to converge to other divergences or distances on the underlying probability distributions (Arjovsky et al. (2017) ; Nowozin et al. (2016) ; Mao et al. (2017) ). Regularization techniques like Gradient Penalty (Gulrajani et al. (2017) ) and Spectral Norm (Miyato et al. (2018) ) did improve the stability of the training process (Kurach et al. (2018) ) but it is still unclear how well this NNs actually approximate such distances even for trivial problems (Pinetz et al. (2018) ). Additionally, it is not at all clear how the generated distribution or the actual target distribution look like. Arora & Zhang (2017) used the birthday paradox to empirically gauge the size of the support of the generated distribution.

GANs are used to transform a well understood low dimensional distribution (in practice either gaussian or uniform) to a high dimensional unknown distribution (Goodfellow et al. (2014) ) by playing a min-max game between two NNs. This paper is based around the intuition, that an estimated probability distribution from a dataset X has high precision if a high percentage of the actual data samples are included in the estimated distribution. To have a sense of what an adequate capacity for a generator network is, we use AE to reconstruct the dataset first. This work relies on the assumption that it is easier to reconstruct the data samples alone, than to reconstruct the entire manifold and Section 5 shows empirical evidence for this. Based on our intuition the manifold consists of the data samples and imposes additional structure between the data samples. In contrast by just reproducing the data samples, no such additional restrictions are given, making the problem strictly simpler. AEs can be trained rapidly and have been researched in detail for a long time (Rumelhart et al. (1985) ). In contrast, trying to do a hyperparameter search on the GAN networks themselves gives rise to all kinds of problems, like instabilities in the training process, random failures and dependence on random seeds for their performance (Lucic et al. (2018) ).

Hence, our contributions are as follows:

• An investigation of the impact of the dimensionality of the latent space on the generated manifold. We showcase that the fit of the data depends heavily on the latent space. We also show similar results thereby to the perception-distortion tradeoff (Blau & Michaeli (2018) ), where with a small dimension for the latent space, the GAN optimizes for perception and the AE optimizes for distortion. • Relating the GAN problem to a compression task and furthermore using compression tools via deep learning to produce a lower bound for a dataset dependent suitable dimensionality of the latent space. • An investigation of the generated manifold and the limitations thereof to produce shifted or noisy images and how this relates to the size of the latent space and overfitting of the generative model.

The remainder of this paper is organized as follows. Section 2 shows the related work. Then in Section 3 we revisit the theory behind pseudo inverting NNs and we explain our methodology in Section 4. In Section 5 the results are shown. Section 6 draws conclusions based on those results.

2. RELATED WORK:

GANs have been invented by Goodfellow et al. (2014) for the task of image synthesis and semisupervised learning. The initial formulation has been shown to suffer from mode collapse and vanishing gradient problems and multiple other loss functions have been proposed (Nowozin et al. (2016) ; Mao et al. (2017) ; Arjovsky et al. (2017) . We choose the Wasserstein GAN, due to it using the L2-distance as its transport cost function to judge the quality of the generated manifold, which is similar to the Mean Squared Error (MSE) used in the classical AE (Rumelhart et al. (1985) ).

To satisfy the Lipschitz constraint we use Gradient Penalty Gulrajani et al. (2017) , due to its good performance in review works (Lucic et al. (2018) ; Kurach et al. (2018) ).

For evaluation purposes we need a quality metrics for GAN and the most common ones rely on the usage of classification models, which are trained on ImageNet (Deng et al. (2009) ). Two commonly used metrics are the Inception Score (Salimans et al. (2016) ) and the Fréchet Inception Distance (FID) (Heusel et al. (2017) ). To combat the bias of having trained on ImageNet, Shmelkov et al. (2018) proposed to train a new classifier to evaluate the GAN model, but this is time consuming and prone to errors by not having a standardized metric. We opted for the FID score, due to its prevalence in related works (Lucic et al. (2018) ; Kurach et al. (2018) ).

AEs have been studied extensively in the past and initially have been proposed by Rumelhart et al. (1985) . They have found usage in compression literature (Cheng et al. (2018a; b) ; Li et al. (2019) ).

Their first usage in a GAN framework has been shown in the Energy based GAN by Zhao et al. (2016) . Since then they have been used as preprocessing to reduce the dimensionality of the data (Guo et al. (2019) ), as regularizers (Che et al. ( 2016)) or as generative models themselves (Webster et al. (2019) ).

The inversion of generator networks has been initially used by Metz et al. (2016) and there have been multiple works on how to improve their method (Creswell & Bharath (2018); Lipton & Tripathi (2017) ) by adding heuristics like stochastic clipping on top of that. We opted against stochastic clipping, because we already project back to the likely set of candidates. Stochastic clipping would add even more randomness to the results. Generalization of GANs through the detection of overfitting has been proposed recently by Webster et al. (2019) . Therein the authors are mostly concerned with detecting memorisation of the training set in the manifold and avoid said memorisation to generalize. In contrast we make the assumption that an estimated probability distribution based on samples should include those samples and overfitting is detected based on a hold out set as well as the ability to reconstruct that hold out set as well as the training set. Additionally, we show that the ability of the GAN to memorize depends mostly on the dimensionality of the latent space and the capacity of the generator network and can be estimated using compression techniques.

3. PRELIMINARIES: WHEN ARE NNS INVERTIBLE?:

In this work, we want to show the fit of the distribution of trained GAN newtworks using inversions of such networks. This motivates the question of "When are NNs invertible?". Due to the nonconvex and non-bijective nature of NNs using ReLU activations invertibility is not given in the general case. The function induced by the NN is in general neither injective nor surjective and it is trivial to construct such cases Arora et al. (2015) .

However, if the weights are gaussian distributed, proofs exist, which guarantee that the NN is invertible with a high probability. For this, we revisit the inversion theory of NNs largely pioneered by Arora et al. (2015) . They empirically showed that the weights of trained NNs actually exhibit random like properties, enabling the usage of Random Matrix Theory (RMT) to investigate such networks. Theorem 1 in (Ma et al. (2018) ) proves that we can invert 2-layer NNs with high probability. In their work they empirically showed that this also holds for multi layer networks. In Figure 1 we show that the same holds true for trained GAN networks. Note, that this property holds across datasets and latent spaces used to train those networks. Glorot initialization (Glorot & Bengio (2010) ) was used to initialize these networks. Therefore, the standard deviation used for the initialization process differs for each layer. However, as is demonstrated in Fig. 1 , all layers converged to a similar gaussian distribution.

Another indicator, that the optimization process is likely to succeed is given by a trick proposed by Li et al. (2018) for classification networks. It allows us to visualize the loss surface of the optimization problem for one sample at a time. For a particular data point z * , we chose two orthogonal vectors α, β and plot the following 2D function:

EQUATION

We chose α = z * -z 0 to model the optimization path towards the original data point. For this experiment we do not need to run an optimization, because the point is to show that it is possible 

4. ESTIMATING A GOOD GAN MODEL FOR A DATASET:

Currently, designing new GAN networks works by running a huge hyperparameter search on a plausible set of combinations (Lucic et al. (2018) ; Kurach et al. (2018) ). Lucic et al. (2018) noted that the progress in GANs has actually been quite minimal and vastly overstated, and depends on the hyperparameters more than on the improvements in the methodology, while Kurach et al. (2018) did notice that non-saturating loss in combiniation with spectral norm produced consistently good results, but never the best ones.

4.1. LOWER BOUND ON THE GENERATOR NETWORK:

The goal of the GAN framework is to encapsulate the high dimensional image distribution in a lower dimensional noise distribution, whereas a generator network maps the lower dimensional distribution to the former high dimensional image distribution (Goodfellow et al. (2014) ). This implies that we compress the larger distribution onto the simple lower dimensional distribution and use the generator network as an approximation for the decompression. We rely on the assumption, that less capacity is needed to reconstruct the training set, than to reconstruct the entire manifold. Therefore, to gain some insight into the necessary capacity of the generator network, the usage of an AE is proposed to gauge the number of parameters and the size of the latent space to enable suitable approximation of the training set. In theory, a latent dimension of 1 would be sufficient to reconstruct the entire dataset, by mapping a curve to the datapoints. A simple smooth function that does this for a dataset {x 1 , .., x n } = X ∈ R n×d and {p 1 , ..., p n } = P ∈ R n×1 looks like this:

EQUATION

This function will output the corresponding image x i for every p i . In practice, this behavior is not observed. AEs with small latent spaces tend to output blurry approximations of the training images as is shown in Section 5. Once a suitable AE for a dataset is found, the decoder part of the AE is used as the generative model of choice. The AE is trained using the L2-loss, which also is the transport cost function used in WGANs (Arjovsky et al. (2017) ) to gauge the quality of the generated manifold. Therefore, the objective function of the AE using a decoder d and an encoder e is defined as follows:

L = min d,e

||d(e(x)) -x|| 2 (3)

We do not opt for any regularization, because we want to fit the data as closely as possible to determine if the generator capacity is adequate for reconstructing the dataset. In this work we show empirically in Sec. 5 that if the AE already fails to produce crisp results, then also the GAN algorithm fails at having the dataset included in the generated manifold. One thing to note is that, while the encoder part is used as our discriminator, it has a completely different objectives and it is unclear, if a better encoder is also a better discriminator.

4.2. DETERMINING THE FIT OF THE DATA IN GANS:

The key part of this work is to analyse the ability of the GAN to memorise the training dataset, following the assumption, that a probablity distribution based on a sample dataset has to include that dataset. To our best knowledge there is no analytical way to invert generative NNs. Therefore this ill posed problem is stated as an inverse optimzation problem for a function g:

z * = arg min z ||g(z) -x|| 2 2 (4)

This is solved using a standard first order gradient method. However, as is shown in Section 5 the resulting latent vectors are unlikely to stem from the targeted noise distribution, if there are no constraints added. Therefore, the search space for z is restricted to the space of plausible latent vectors. Plausibility in this context means that the probability of the latent vector originating from this space is larger than 99.7% (3-sigma-rule). In the standard multivariate gaussian distribution the norm of the vector z corresponds inversely to the likelihood of z ∈ N (0, I). Therefore we restrict the norm to be ||z|| ≤ E[||z||] + 3 V AR [||z||] . For the multivariate gaussian distribution this is easy to calculate and the final objective function is given as follows:

EQUATION

This is solved using a projected ADAM optimizer (Kingma & Ba (2014)).

5. EXPERIMENTS:

Our experiments are done on medium sized datasets (CIFAR-10 and CelebA (64 × 64 resolution)) and the proof of concept on MNIST is relegated to the supplementary material. The NN adapted for this task is the standard DCGAN from Salimans et al. (2018) . Convolutional layers and resize operations are used instead of transposed convolutions, to avoid checkerboard artifacts (Odena et al. (2016) ). The full hyperparameter setting is shown in the supplementary material. The visual quality measured by the FID (Heusel et al. (2017) ) of the resulting samples is on par with the survey work by Lucic et al. (2018) . This work is not aimed at improving the visual quality of the samples generated by GAN algorithms. For CIFAR-10 the test set is used as our validation set and for CelebA we randomly split off 20% of the dataset to be used as validation set.

5.1. RELATIONSHIP BETWEEN AE AND GAN RECONSTRUCTION LOSS:

In principle, getting from an AE to a GAN is just a rearrangement of the NNs. For the GAN training, the default hyperparameter are used as in Gulrajani et al. (2017) . The AE NNs are optimized using the ADAM(Kingma & Ba ( 2014)) optimizer with default hyperparameters for all the datasets, always on a single GPU. A comparison of the resulting reconstruction errors is shown in Fig. 3 . Therein the different AE and GAN models only differ in their latent space sizes. Increasing the latent space dimension also increases the number of parameters. However, for the sake of simplicity this was neglected. The AE network acts as a lower bound for the GAN algorithm, therefore validating our intuition that the AE complexity lower-bounds the GAN complexity.

5.2. VISUAL QUALITY OF GANS VS AE GENERATED IMAGES:

The visual quality of the generated images is evaluated using the FID (Heusel et al. (2017) ) versus the reconstructed images by an AE for the same latent space sizes. The experimental results are shown in Fig. 4 for the CIFAR and CelebA datasets and demonstrate, that the visual quality of the GAN generated images barely changes with different latent dimensionality. In contrast, the AE reconstructions gain quality and sharpness as the latent space increases. This reflects the output of those models shown in Fig. 5 , where the resulting images are clearly an approximation of the actual training images. On the other hand, the GAN reconstructions have similar visual quality independent of the actual latent space. However, the resulting images are only remotely similar to the actual image for insufficiently large latent spaces. This phenomenon has been observed in other works, where a similar looking person to an actual training image is found by the inversion procedure (Webster et al. (2019) ). However, we show in Fig. 5 , that this phenomenon depends mostly on the size of the latent space. 2017)) curve for different latent spaces (lower is better). Increasing or decreasing the latent space does not result in a noticeable difference in the quality metric for the WGAN-GP, but it does for the AE.

5.3. WHAT IS INCLUDED IN THE GENERATED MANIFOLD?:

While it is possible to reconstruct the dataset accurately using a large enough latent space, in this section we look at what else those models can reconstruct. In these experiments, our baseline model Figure 6 : Impact of translation on the reconstruction ability of GAN networks. Notice, that by increasing latent space dimension, the generator network is able to reconstruct translations with a similar precision as the original images.

The second experiment is on reconstructing the CIFAR-10 dataset using the CelebA trained model.

To provide the same resolution, the CIFAR-10 images were upscaled to 64 × 64 using bilinear interpolation. If the latent space is large enough, we can reconstruct the images well, as shown in Fig. 7 . The reconstruction of the CelebA images is always worse as measured by the MSE (Fig. 7 ).

The final experiment is on reconstructing noise images. For this a dataset of uniform noise images with size 64 × 64 is used. The results are shown in Fig. 8 . As is suggested by the results, it is in fact not possible to reconstruct everything. One thing to note though is that with lower latent spaces some face structure is visible in the reconstructed images, showing that those models have a strong prior on the face structure. We noticed and the corresponding experiment is shown in the supplementary material, that the norm of the latent vector is smaller than of random vectors. Our hypothesis is that those images are the base face features, which are refined with larger latent vectors, but this guess needs further validation work. 

6. CONCLUSIONS:

In this work, we show that by reducing the problem to a compression task, we can give a lower bound on the required capacity and latent space dimensionality of the generator network for the distribution estimation task. Relating these two different algorithms to each other, the literature surrounding AEs for invertability and dimensionality reduction, as well as the corresponding theoretical advancements are used. While in this work the encoder and the discriminator NNs use the same architecture, we have not discovered any relation between them. Still, the same architecture works well empirically for both task.

Using our framework we show various properties of generator networks. The perceptual image quality appears to be independent of the actual size of the latent space, which is in contrast to AE, where the visual quality improves if the dimensionality of the latent space is increased. However, the ability to reconstruct the training set correctly does depend on the initial latent space. Also the ability of the generator to reconstruct deviations from the original dataset, like a validation set or shifted images depends just as much on the initial latent space. However, the same cannot be said for reconstructing arbitrary noise images. Here the reconstruction ability is independent of the initial latent space unless it is chosen very large, suggesting that the generator has learned realistic natural image features. Here for smaller latent spaces we can still observe face like features. Our hypothesis is that the implicit bias induced by the generator architecture lends itself to generating natural images and GANs are skilled at that by learning primitives which can be combined to construct arbitrary images. In future works we want to use our setup to search towards better and more reliable generators for images. The initial experiments on MNIST validating the legitimacy of the approach are shown in Fig. 9 . Instead of using a visual quality metric to determine if the resulting images are similar enough, we used the reconstructed training images of the GAN for training a LeNet network LeCun et al. (1998) to determine if the images are descriptive. The manifold of MNIST has been faithfully learned before by Shmelkov et al. (2018) . In contrast to their work, we show that this only works given a large enough latent space and that the reconstruction error of the AE and the WGAN-GP converge for larger latent spaces. 

:



Figures and Tables:

Figure1: Histogram of the values in the weight matrix and the convolutional layers of a trained WGAN-GP with a latent space specific dimensionality for CIFAR-10 and CelebA datasets. Notice, that the weights share the same characteristics as if they were drawn from a normal distribution.

Figure 2: Loss surface around a data point of the generator of a WGAN-GP trained on CelebA.There is a clear optimization path from z 0 = 0 to a point z * even in this simplified plot, empirically validating the claims ofLipton & Tripathi (2017), that an accurate reconstruction of the generator manifold is possible works using first order methods.

Figure3: Impact of the latent space on the AE and GAN reconstructions loss on CIFAR-10 and CelebA. The AE acts as a lower bound for the GAN reconstruction loss and the reconstruction loss decreases for both the AE and the GAN generator with increasing latent space dimensions.

Figure 4: FID(Heusel et al. (2017)) curve for different latent spaces (lower is better). Increasing or decreasing the latent space does not result in a noticeable difference in the quality metric for the WGAN-GP, but it does for the AE.

Figure5: The first columns show the original images. As we increase the latent space of the autoencoder the image quality increasing. The GAN reconstructions are of high perceptual quality even for low dimensional spaces, but correspond to other people as has been observed byWebster et al. (2019).

Figure 7: Left & Right: Reconstruction of CIFAR images in a WGAN-GP model trained on CelebA images.As the latent dimensionality is increased, the reconstruction is improved for both CIFAR-10 and CelebA. Right: With a small latent space, the face image features are retained, while with a large latent space, the CIFAR-10 images are faithfully reconstructed.

Figure 9: Mnist GAN and AE reconstructions in (a). Visual quality metrics are not really applicable on MNIST data so in (b) the reconstructed GAN images are used to train a LeNet to classify on MNIST and is compared to training on the true training set.

Architecture of the Discriminator and Generator networks. Initialization is done usingGlorot & Bengio (2010) PROOF OF CONCEPT ON MNIST

Table content: <table/>

Peer review (iclr2020):
The work performs a systematic empirical study of how the latent space design in a GAN impacts the generated distribution. The paper is well-written and was easy to read. Also, I find this to be an interesting and promising direction of research. The convergence proof in Goodfellow (2014) assumes that the data distribution has a density, and essentially states that the JS-divergence is zero if and only if the two distributions are the same. In practice, the data distribution is discrete, while the latent distribution has a probability density function. It is not possible to transform a density into a discrete distribution by a continuous map and neural networks are always continuous by construction. In theory, as training progresses, more and more latent mass will be pushed on the discrete samples and no minimizer exists (unless the function space of generator is constrained or the real distribution is smoothed out a bit). Since it is not possible to assess whether the GAN training has converged due the nonconvexity of the energy and non-existence of a global optimizer, the empirically observed results might be very specific to the chosen optimization procedure, stopping criterion, dataset, hyper-parameters, initialization, network architectures, etc etc. It is a challenge to study the choice of latent space in a somewhat *isolated* way. These issues should be discussed in the paper and the reader should be made aware of such problems. Another point, could it be, that by increasing the dimension of the latent space, one makes it easier for the nonconvex optimization in (5) to converge to *unlikely but realistic looking samples*? I think this is not too far-fetched, as increasing the dimension of an optimization problem often makes local optimization less likely to get stuck at local optima. Also it might not be the best idea to optimize (5) with Adam since it is not a stochastic optimization problem and there are provably convergent solvers out there for this problem class. Since it is possible to evaluate the likelihood of the optimized reconstructions that are nearby the data points, one could check whether this is indeed the case. While constrained not to be too unlikely, I wonder whether the likelihood increases or decreases with the dimensionality of the latent space and this would make an interesting plot. Unfortunately, I did not understand the connections to auto-encoders, as they might optimize a fundamentally different criterion than GANs. In particular *In principle, getting from an AE to a GAN is just a rearrangement of the NNs. * is unclear to me. Also, what is meant by lower-bound? Is the claim that the reconstruction error in an auto encoder will be lower, than if one optimizes the latent code in a GAN to reconstruct the input? Figure 3 seems to support this hypothesis, but I don*t have an intuition why this should be true and have some doubts. A mathematical proof seems out of reach. I have trouble to understand the *intuition that the AE complexity lower-bounds the GAN complexity.* Before reading this paper, my intuition was the opposite: If the generator distribution covers the real distribution, the reconstruction error for GAN is zero. Intuitively, it seems a much easier task to somehow cover a distribution than to minimize an average reconstruction error. The connection of WGANs to the L2 reconstruction loss in the auto-encoder is very hand-wavy. It is still an open question whether WGANs actually have anything to do with the Wasserstein distance. People working in optimal transport doubt this, due to huge amount of approximations going on. At this point I*m reluctant to recommend acceptance, as the paper tries to connect things, which for me are quite disconnected and the evaluations of reconstruction error, etc. might depend in intricate ways on the nonconvex optimization procedures. Minor suggestions, typos, etc (no influence on my rating): - What is the *generated manifold* that is talked about in the introduction, contributions and throughout the paper? To me, it is not directly clear that the support of the transformed distribution will be a manifold (especially if G is non-injective). Anyway, the manifold structure is nowhere exploited in the paper, so I suggest to call it *transformed latent distribution*. - Had to pause a little bit to understand Eq. 2 (simple polynomial interpolation). It is unnecessary to show the explicit form, as I*m sure no one doubts the existence of a smooth curve interpolating a finite set of points in R^d. - Equations should always include punctuation marks. - Eq. 5: dim --> 	ext{dim} and s.t. --> 	ext{s. t.} - Fig 3b: the red curve is missing or hidden behind another curve.