Impact of the latent space on the ability of GANs to fit the distribution
Authors: Thomas Pinetz
Daniel Soukup
Thomas Pock
Venue:  (2020)
Keywords: Deep Learning
Generative Adversarial Networks
Compression
Perceptual Quality

Abstract: The goal of generative models is to model the underlying data distribution of a
sample based dataset. Our intuition is that an accurate model should in principle
also include the sample based dataset as part of its induced probability distribution.
To investigate this, we look at fully trained generative models using the Generative
Adversarial Networks (GAN) framework and analyze the resulting generator
on its ability to memorize the dataset. Further, we show that the size of the initial
latent space is paramount to allow for an accurate reconstruction of the training
data. This gives us a link to compression theory, where Autoencoders (AE) are
used to lower bound the reconstruction capabilities of our generative model. Here,
we observe similar results to the perception-distortion tradeoff (Blau & Michaeli
(2018)). Given a small latent space, the AE produces low quality and the GAN
produces high quality outputs from a perceptual viewpoint. In contrast, the distortion
error is smaller for the AE. By increasing the dimensionality of the latent
space the distortion decreases for both models, but the perceptual quality only
increases for the AE.

Under review as a conference paper at ICLR 2020
IMPACT OF THE LATENT SPACE ON THE ABILITY OF GANS TO FIT THE DISTRIBUTION
Anonymous authors Paper under double-blind review
ABSTRACT
The goal of generative models is to model the underlying data distribution of a sample based dataset. Our intuition is that an accurate model should in principle also include the sample based dataset as part of its induced probability distribution. To investigate this, we look at fully trained generative models using the Genera- tive Adversarial Networks (GAN) framework and analyze the resulting generator on its ability to memorize the datasct. Further, we show that the size of the initial latent space is paramount to allow for accurate reconstruction of the training data. This gives us a link to compression theory, where Autoencoders (AE) are used to lower bound the reconstruction capabilities of our generative model. Here, we observe similar results to the perception-disee distortion tradeoff (Blau & Michaeli (2018)). Given small latent space, the AE produces low quality and the GAN produces high quality outputs from a perceptual viewpoint. In contrast, the dis- tortion error smaller for the AE. By increasing the dimensionality of the latent space the distortion decreases for both models, but the perceptual quality only increases for the AE
INTRODUCTION
Generative Adversarial Networks (GANs) were introduced by Goodfellow al. (2014) for the pur- pose of generative modelling. Since then this framework has been successfully applied to works in style transfer by Karras et al. (2018), superresolution by (hhcher al.] (2018) and semi-supervised learning by Salimans et al. (2016). but what GANs actually learn is still poorly understood as has been noted by Webster et al. (2019). Recently, GANs have been used to solve inverse problems, where it was tried to use the generated manifold to solve an auxiliary task like image completion (Webster et al. (2019), MRI reconstruction (Narnhofer et al. (2019) or anomaly detection (Shocher cet (2018). For those applications, it is necessary to know if the generator NN actually describes the distribution well. Related works have shown that faithfully reconstructing the images from a generator network is not trivial (Webster al. (2019); Shmelkov et al. (2018)).
The original convergence proof by Goodfellow et al. (2014) assumes that the generator and discrim- inator Neural Networks (NNs) have infinite capacity and they showed that the diseriminator network models the Jensen-Shannon divergence between the probability distribution induced by the gener- ator and the real data distribution. Others have adapted this paradigm and devised loss functions which have been shown to converge to other divergences or distances on the underlying probabil- ity distributions (Arjovsky et al. (2017): Nowozin et al. (2016); Mao al. (017)). Regularization techniques like Gradient Penalty (Gulrajani et al. (2017) () and Spectral Norm (Miyato et al. (2018) did improve the stability of the training process (Kurach
al. (2018) ) but it is still uncl ear how well this NNs actually approximate such distances even for trivial problems (Pinetz et al. (2018)
Additionally, it is not at all clear how the generated distribution or the actual target distribution look like. Arora& & Zhang of the generated distribution.
(2017) used the birthday paradox to empirically gauge the size of the support
GANs are used to transform well understood low dimensional distribution (in practice either gaus- sian or uniform) to high dimensional unknown distribution (Goodfellow et al. (2014)) by playing min-max game between two NNs. This paper is based around the intuition, that estimated probability distribution from a dataset X' has high precision if a high percentage of the actual data samples included in the estimated distribution. To have sense of what adequate capacity for

---

Under review as a conference paper at ICLR 2020
generator network is, use AE reconstruct the dataset first. This work relies on the assump- tion that it is easier to reconstruct the data samples alone, than to reconstruct the entire manifold and SoctionESssowee empirical evidence for this. Based on intuition the manifold consists of the data samples and imposes additional structure between the data samples. In contrast by just reproducing the data samples, no such additional restrictions are given, making the problem strietly simpler. AEs can be trained rapidly and have been researched in detail for long time (Rumelhart et al.) (1985) ). In contrast, trying do a hyperparameter search on the GAN networks themselves gives rise to all kinds of problems, like instabilities in the training process, random failures and dependence on random seeds for their performance (Lucic al. (2018))
Hence, our contributions are as follows:
An investigation of the impact of the dimensionality of the latent space on the generated manifold. We showcase that the fit of the data depends heavily on the latent space. We also show similar results thereby to the perception-distorte where with small dimension for the latent space, the GAN optimizes for perception and the AE optimizes for distortion.
tradeofi & Michaeli (2018),
Relating the GAN problem compression task and furthermore using compression tools via deep learning to produce a lower bound for dataset dependent suitable dimensionality of the latent space.
An investigation of the generated manifold and the limitations thereof to produce shifted or noisy images and how this relates to the size of the latent space and overfitting of the generative model.
The remainder of this paper is organized as follows. Section 2 shows the related work. Then in Section 3 we revisit the theory behind pseudo inverting NNs and we explain methodology in Section 4. In Section 5 the results are shown. Section 6 draws conclusions based on those results.
RELATED WORK
GANs have been invented by (Goodfellow et al. (0014) for the task of image synthesis and semi- supervised learning. The initial formulation has been shown to suffer from mode collapse and vanishing gradient problems and multiple other loss functions have been proposed (Nowozin et al. (201)); Mao et al. 2017):Ba AAjovsky et al. (2017). We choose the Wasserstein GAN, due to it using 1.2-distance as transport cost function to judge the quality of the generated manifold, which is similar to the Mean Squared Error (MSE) used in the classical AE (Rumelhart et al. (1985) ). To satisfy the Lipschitz constraint we use Gradient Penalty Guulaaaai et al. (2017). due to its good performance in review works(Lucie al. (2018); Kurach et al. (2018)).
For evaluation purposes we need a quality metrics for GAN and the most common ones rely on the usage of classification models, which are trained on ImageNet (Deng et al. (2009)). Two commonly used metrics are the Inception Score (Salimans et al. (2016)) and the Frechet Inception Distance (FiD)(Heusel et al. (2017)). To combat the bias of having trained ImageNet. (Shmelov et al.) 2018) proposed to train new classifier to evaluate the GAN model, but this is time consuming and prone to errors by not having a standardized metric. We opted for the FID score, due to its prevalence in related works works(Lucic et al. (2018); Kurach et al. (2018)).
AEs have been studied extensively in the past and initially have been proposed by Rumelhart et al. (1885.. They have found usage in compression literature (Cheng et al. (01 8a b): Li al. (2019). Their first usage
GAN framework has been shown in the Energy based GAN by Zhao et al.
(016)). Since then they have been used as preprocessing to reduce the dimensionality of the data
(Guo et al. (2019), as regularizers (Che et al. (2016)) or as generative models themselves (Webster
ct al. (2019))
The inversion of generator networks has been initially used by Metz et al. (016) and there have been multiple works on how to improve their method (Creswell & Bharath (2018) Lipton & Tri- pathi (2017) by adding heuristics like stochastic clipping on top of that. We opted against stochastic clipping, because already project back to the likely set of candidates. Stochastic clipping would add even more randomness to the results. Generalization of GANs through the detection of overfit- ting has been proposed recently by Webster al. (2019). Therein the authors are mostly concerned
2

---

Under review as a conference paper at ICLR 2020
bpprr Lepet
Layerd Leyor5
Lapsr
Lopat
(a) CIFAR-I0 (IFAA-10:dim(a) = 10
(b) CIFAR-10: dim(2) = 100
(c) CIFAR- 10: dim(z)) 2000
7 x
Layer
Loper
Lapat
(d) CelebA: dim(z) = 10
(e) CelebA: dim(=) = 100
(1) CelebA: dim(2= = 5000
Figure 1: Histogram of the values in the weight matrix and the convolutional layers of trained WGAN-GP with a latent space specific dimensionality for CIFAR-10 10 and CelebA datasets. Notice, that the weights share the same characteristics as if they were drawn from a normal distribution.
with detecting memorisation of the training in the manifold and avoid said memorisation to generalize. In contrast we make the assumption that an estimated probability distribution based on samples should include those samples and overfitting detected based on a hold out set as well as the ability to reconstruct that hold out set as well as the training set. Additionally, we show that the ability of the GAN memorize depends mostly on the dimensionality of the latent space and the capacity of the generator network and can be estimated using compression techniques.
PRELIMINARIES, WHEN ARE NNS INVERTIBLE?
In this work, want show the fit of the distribution of trained GAN newtworks using inversions of such networks. This motivates the question of "When are NNs invertible?". Due to the non- convex and non-bijective nature NNs using ReLU activations invertibility is not given in the general case. The function induced by the NN is in general neither injective nor surjective and it is trivial to construct such cases Arora al. 2015).
However, if the weights are gaussian distributed, proofs exist, which guarantee that the NN is in- vertible with high probability. For this, we revisit the inversion theory of NNs largely pioneered by |Arora et al. (2015). They empirically showed that the weights of trained NNs actually exhibit random like properties, enabling the usage of Random Matrix Theory (RMT) to investigate such networks. Theorem 1 in (Ma et al. (20T8)) proves that can invert 2-layer NNs with high prob- ability. In their work they empirically showed that this also holds for multi layer networks. In we we show that the same holds true for trained GAN networks. Note, that this property holds across datasets and latent spaces used to train those networks. Glorot initialization (Glorot & Bengio 2010 used to initialize these networks. Therefore, the standard deviation used for the initial- ization process differs for each layer. However, as is demonstrated in Fig. [ all layers converged to similar gaussian distribution.
Figure [
Another indicator, that the optimization process is likely to succeed is given by trick proposed by) Li al. (2018) for classification networks. allows us to visualize the loss surface of the optimization problem for one sample at time. For a particular data point 2", we chose two orthogonal vectors a, (3 and plot the following 2D function:
f(a. b) = 1(2" + aa + 63)
We chose a = 2n 20 to model the optimization path towards the original data point. For this experiment do not need optimization, because the point is to show that it is possible
3

---

Under review as a conference paper at ICLR 2020
100
1.
4.3
0.5
0.6
0.75
04
0.4
(a) dim(z) 10
(b) dim(2) 100
(c) dim(z) 1000
Figure 2: Loss surface around data point of the generator of a WGAN-GP trained on CelebA. point even in this simplified plot, empirically
There is clear optimization path from 20 validating the claims of Lipton & Tripathi (2017), that an accurate reconstruction of the generator manifold is possible works using first order methods.
recover points from the manifold. To ensure that is orthogonal vector, we simply chose this vector by setting 3 = and then 8[0] = a[1] and 3[] = -a(0]. Then a, 3 are orthogonal and them basis for the plots shown in Fig.
scaled to have the same norm as a; e.g. || ||a||2. We vary a, b E [-2. 2]. In our experiments this it is trivial to recover points from the manifold and impossible to recover the training images, than this implies that the manifold does not include the training images.
For visualization purposes,
always resulted benign structures as shown in Fig.
4 ESTIMATING GOOD GAN MODEL FOR A DATASET
Currently, designing new GAN networks works by rumning a huge hyperparameter search on plausible set of combinations (Lucic al. (2018); Kurach al. (018)). Lucic al. (2018) noted that the progress in GANs has actually been quite minimal and vastly overstated, and depends on the hyperparameters more than on the improvements in the methodology, while Kurach et al. (2018) did notice that non-saturating loss in combiniation with spectral norm produced consistently good results, but never the best ones.
4.1 LOWER BOUND ON THE GENERATOR NETWORK
The goal of the GAN framework is to encapsulate the high dimensional image distribution lower dimensional noise distribution, whereas generator network maps the lower dimensional distribution to the former high dimensional image distribution (Goodfellow et al.] (2014)). This implies that compress the larger distribution use the generator network as an approximation for the decompression. We rely on the assumption, that less capacity is needed reconstruct the training set, than to reconstruct the entire manifold. Therefore, to gain some insight into the necessary capacity of the generator network, the usage of an AE is proposed gauge the number of parameters and the size of the latent space enable suitable approximation of the training set. In theory, a latent dimension of would be sufficient to reconstruct the entire dataset, by mapping a curve to the datapoints. A simple smooth function that does this for dataset (II,.., = X E Rnxdd and (pI,..,,,, P [Rax1 looks like this:
the simple lower dimensional distribution and
n
71
(P;
g(2)
3
(p; Ps)
j=1ij
This function will output the corresponding image F; for every Pi- In practice, this behavior is not observed. AEs with small latent spaces tend to output blurry approximations the training images is shown in Section Once suitable AE for dataset is found, the decoder part of the AE
4

---

Under review as a conference paper at ICLR 2020
is used as the generative model of choice. The AE is trained using the L2-loss, which also is the transport cost function used in WGANs (Arjovsky et all (2017)) to gauge the quality of the generated manifold. Therefore, the objective function of the AE using decoder d and encoder is defined as follows:
L = min [[d(e(=))))))))))))))))))))))))) - x:12 d,e
(3)
We do not opt for any regularization, because we want fit the data as closely possible to determine if the generator capacity is adequate for reconstructing the dataset. In this work we show empirically in Sec. [ that if the AE already fails to produce crisp results, then also the GAN algorithm fails at having the dataset included in the generated manifold. One thing to note is that, while the encoder part is used as our discriminator, has a completely different objectives and unclear, if better encoder is also better discriminator.
4.2 DETERMINING THE FIT OF THE DATA GANs
The key part of this work is to analyse the ability of the GAN to memorise the training dataset, following the assumption, that probablity distribution based on sample dataset has include that dataset. To our best knowledge there is no analytical way to invert generative NNs. Therefore this ill posed problem is stated as an inverse optimzation problem for function g:
arg min [9(2) x012
This solved using a standard first order gradient method. However, as is shown in Section the resulting latent vectors are unlikely to stem from the targeted noise distribution, if there are no constraints added. Therefore, the search space for = restricted to the space of plausible latent vectors. Plausibility in this context means that the probability of the latent vector originating from this space is larger than 99.7% (3-sigma-rule). In the standard multivariate gaussian distribution the norm of the vector corresponds inversely the likelihood of (N, Z). Therefore restrict the norm be ||=|| E[||=[|] + easy to calculate and the final objective function is given as follows:
For the multivariate gaussian distribution this is
2" = arg min [1(2)) 101
(5)
s.t. || S dim(z) 3 2dim(22)
This is solved using projected ADAM optimizer (Kingma & Ba 2014)
EXPERIMENTS
Our experiments done medium sized datasets (CIFAR-10 10 and CelebA (64 x 64 resolution)) and the proof concept on MNIST is relegated to the supplementary material. The NN adapted for this task is the standard DCGAN from SSalimans al. (2018). Convolutional layers and resize operations are used instead of transposed convolutions, to avoid checkerboard artifacts (Odena et al.) (2016). The full hyperparameter setting is shown in the supplementary material. The visual quality measured by the FID (Heusel et al. (2017)) of the resulting samples is on par with the survey work by Lucic et al. (2018). This work is not aimed improving the visual quality of the samples generated by GAN algorithms. For CIFAR-10 10 the test set is used as our validation set and for CelebA we randomly split off 20% of the dataset be used validation set.
RELATIONSHIP BETWEEN AE AND GAN RECONSTRUCTION LOSS
In principle, getting from AE GAN is just rearrangement of the NNs. For the GAN train- ing, the default hyperparameter are used as in Guirajani et all (2017). The AE NNs are optimized using the (Kigma & Ba (014)) optimizer with default hyperparameters for all the datasets,
5

---

Under review as a conference paper at ICLR 2020
WGAN GP Train WGAN-GP Val
WGAN Train
Reconstruction Error
Reconstruction Error
WGAN-GP
10-2
AE Train AE Val
AE Train
AE Val
10
101
dimiz)
dimiz)
(a) Cifar
(b) CelebA
Figure 3: Impact the latent space on the AE and GAN reconstructions loss on CIFAR- 10 and CelebA. The AE acts as a lower bound for the GAN reconstruction loss and the reconstruction loss decreases for both the AE and the GAN generator with increasing latent space dimensions.
always on single GPU. comparison of the resulting reconstruction errors is shown in Fig. Therein the different AE and GAN models only differ in their latent space sizes. Increasing the latent space dimension also increases the number of parameters. However, for the sake of simplic- ity this was neglected. The AE network acts as lower bound for the GAN algorithm, therefore validating our intuition that the AE complexity lower-bounds the GAN complexity.
5.2
VISUAL QUALITY OF GANS VS AE GENERATED IMAGES
The visual quality of the generated images is evaluated using the FID (Heusel al. (2017) versus the reconstructed images by an AE for the same latent space sizes. The experimental results are shown in Fig. for the CIFAR and CelebA datasets and demonstrate, that the visual quality of the GAN generated images barely changes with different latent dimensionality. In contrast, the AE reconstructions gain quality and sharpness as latent space increases. This reflects the output of those models shown in Fig. [ where the resulting images are clearly an approximation of the actual training images. On the other hand, the GAN reconstructions have similar visual quality independent of the actual latent space. However, the resulting images only remotely similar to the actual image for insufficiently large latent spaces. This phenomenon has been observed in other works, where similar looking person procedure (Webster et al. (2019)). However, we show in Fig. that this phenomenon depends mostly the size of the latent space.
actual training image is found by the inversion
WGAN
WCAN
2
(a) CIFAR
(b) CELEBA
Figure 4: FID Heusel et al. 2017 curve for different latent spaces (lower is better). Increasing or decreasing the latent space does not result in noticeable difference in the quality metric for the WGAN-GP, but does for the AE.
5.3 WHAT IS INCLUDED IN THE GENERATED MANIFOLD?
While it is possible to reconstruct the dataset accurately using large enough latent space, in this section look what else those models can reconstruct. In these experiments, our baseline model

---

Under review as a conference paper at ICLR 2020
dimizi
dimit
(a) AE
(b) WGAN-GP GP
Figure 5: The first columns show the original images. As we increase the latent space of the autoen- coder the image quality increasing. The GAN reconstructions are of high perceptual quality even for low dimensional spaces, but correspond to other people as has been observed by Webster et al. (2019).
is WGAN-GP trained on CelebA. The first experiment shown in Fig. 6 uses translated training images. This is especially challenging for this dataset, due to the pre-aligned face images used the CelebA dataset. As is shown in Fig. the error goes up and the visual quality of the reconstructions deteriorates even for small pixel shifts as long as the latent space small enough.
dimiz] 100
GT
dimiz) - 500
dimiz) 5000
dlm(z)
500
5000
Translation in Pixel
Translations in Pixel
Figure 6: Impact of translation on the reconstruction ability of GAN networks. Notice, that by increasing latent space dimension, the generator network is able to reconstruct translations with similar precision as the original images.
The second experiment on reconstructing the CIFAR-10 dataset using the CelebA trained model. To provide the same resolution, the CIFAR-10 images were upscaled to 64 x 64 using bilinear interpolation. If the latent space large enough, we can reconstruct the images well, as shown in Fig. 7 The reconstruction of the CelebA images is always worse as measured by the MSE (Fig. 7
The final experiment on reconstructing noise images. For this dataset of uniform noise images with size 64 x 64 is used. The results are shown in Fig. 8 As is suggested by the results, it is in fact not possible to reconstruct everything. One thing to note though is that with lower latent spaces some face structure is visible in the reconstructed images, showing that those models have strong prior on the face structure. We noticed and the corresponding experiment is shown the supplementary material, that the norm of the latent vector is smaller than of random vectors. Our hypothesis is that those images are the base face features, which are refined with larger latent vectors, but this guess needs further validation work.
7

---

Under review as a conference paper at ICLR 2020
0.200
CIFAR CELEBA
0.150
dimiz)
MSE
6.100
0.075
0.050
500
0.025
1000
50
dim(z)
Figure 7: Left & Right: Reconstruction of CIFAR images in WGAN-GP model trained on CelebA images. As the latent dimensionality is increased, the reconstruction is improved for both CIFAR-10 10 and CelebA. Right: With small latent space, the face image features retained, while with large latent space, the CIFAR-10 images are faithfully reconstructed.
WGAN-GP
0.30
0.25
100
dimiz)
500
0.15
0.10
5000
100
540
1000
5:000
dim
Figure 8: Reconstruction error of uniform noise images between [-1, 1] using generator network trained on CelebA images. With smaller latent spaces, it is still possible to see some key face features. larger latent space manages to produce better reconstructions.
CONCLUSIONS
In this work, we show that by reducing the problem to compression task, we give lower bound on the required capacity and latent space dimensionality of the generator network for the distribution estimation task. Relating these two different algorithms to each other, the literature surrounding AEs for invertability and dimensionality reduction, as well as the corresponding theoretical advancements used. While in this work the encoder and the discriminator NNs the architecture, have not discovered any relation between them. Still, the same architecture works well empirically for both task.
Using our framework we show various properties of generator networks. The perceptual image quality appears be independent of the actual size of the latent space, which is contrast AE, where the visual quality improves if the dimensionality of the latent space is increased. However, the ability reconstruct the training set correctly does depend on the initial latent space. Also the ability of the generator to reconstruct deviations from the original dataset, like validation set or shifted images depends just as much on the initial latent space. However, the same cannot be said for reconstructing arbitrary noise images. Here the reconstruction ability is independent of the initial latent space unless it is chosen very large, suggesting that the generator has learned realistic natural image features. Here for smaller latent spaces we can still observe face like features. Our hypoth- esis is that the implicit bias induced by the generator architecture lends itself to generating natural images and GANs are skilled at that by learning primitives which can be combined to construct bitrary images. In future works we want to use our setup to search towards better and more reliable generators for images.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 223, 2017.
Sanjeev Arora and Yi Zhang. Do gans actually learn the distribution? an empirical study. arXiv preprint 1706v:1706.. 17t.:.022...0.4444 2017.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. Why are deep nets reversible: A simple theory, with implications for training. arXiv preprint ar 2015.
Yochai Blau and Tomer Michaeli. The cnceptiionddistoriin tradeoff. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 0228-6237, 2018.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. arXiv preprint arXiv:1612.02136, 2016.
Zhengxue Cheng. Heming Sun, Masaru Takeuchi, and Jiro Katto. Deep convolutional autoencoder- based lossy image compression. In 2018 Picture Coding Symposium (PCS), pp. 253-257. IEEE, 2018a.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Performance comparison of con- volutional autoencoders, generative adversarial networks and super-resolution for image com- pression. In CVPR Workshops, pp. 2613-2616, 2018b.
Antonia Creswell and Anil Anthony Bharath. Inverting the generator of a generative adversarial network. IEEE transactions on neural networks and learning systems, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedingy of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp- 5769-5779, 2017.
Yong Guo, Qi Chen, Jian Chen, Qingyao Wu, Qinfeng Shi, and Mingkui Tan. Auto-embedding generative adversarial networks for high resolution image synthesis. IEEE Transactions on Mul- timedia, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A bayee generator architecture for generative adversarial networks. arXiv preprint arXiv: 18/2,04949 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 4rXiv:1412.69800 69800 2014.
Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. The gan landscape: Losses, architectures. regularization, and normalization. arXiv preprint arXiv:1807.04720, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEF 86(1 1):227)-22244 1998.

---

Under review as a conference paper at ICLR 2020
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In Advances in Neural Information Processing Systems, pp. 8389-6399, 2018.
Hongfei Li, Lili Meng, Jia Zhang, Yanyan Tan, Yuwei Ren, and Huaxiang Zhang. Multiple descrip- tion coding based on convolutional auto-encoder. IEEE Access,
2019.
Zachary C Lipton and Subarna Tripathi. Precise recovery of latent vectors from generative adver- 2017.
sarial networks. arXiv preprint
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In Advances in neural information processing systems, pp- 698-707, 2018.
Fangehang Ma, Ulas Ayaz, and Sertac Karaman, Invertibility of convolutional generative networks from partial measurements. In Advances in Neural Information Processing Systems, pp. 9628- 1637,2015
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Confer- ence on Computer Vision, pp. 2794-2802, 2017.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. arXiv preprint arXiv: 1et1:0211.... 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. URL[Lttpp::/pppeneevviee www.:/eee/eeeeee
Dominik Narnhofer, Kerstin Hammernik, Florian Knoll, and Thomas Pock. Inverse gans for accel- erated mri reconstruction. Wavelets and Sparsity XVIII, volume 11138, pp. 111381A. Interna- tional Society for Optics and Photonics, 2019.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 279-279, 2016.
Augustus Odena, Vincent Dumoulin, and Chris Olah, Deconvolution and checkerboard artifacts. Distill, 1(10)ee, 2016.
Thomas Pinetz, Daniel Soukup, and Thomas Pock. What is optimized in wasserstein gans? In Proceedings of the 23rd Computer Vision Winter Workshop, 2018
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, In Advances in neural information processing systems, pp. 2234-2242, 2016.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving GANs using optimal transport. In International Conference Learning Representations, 2018. URL httpp:// openreview wwwaarvviewnntt/ooee www.ee/oooomdddoeeeee..
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In Proceedings of the European Conference on Computer Vision (ECCV), pp. 213-229, 2018.
Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani. Internal distribution matching for natural image retargeting. arXiv preprint arXiv:1812.000
2018.
Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. Detecting overfitting of deep generative networks via latent recovery. arXiv preprint arXiv:1901.03396, 2019.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network.
Xixiv preprint 16xiv:1609.03122 2016
10

---

Under review as a conference paper at ICLR 2020
ADDITIONAL MATERIAL
ARCHITECTURE AND HYPERPARAMTER SETTINGS
For our experiments a standard CNN was used as is described in Table [ All the convolutions use x 5 kernels. For the CIFAR-I0 experiments we removed one of the up and downlayers and started with 1024 feature maps. We used the ADAM optimizer with learning rate 1 4 and 31 = O. and 8g = 0.9 similar to the original WGAN-GP as described by Gulrajani et al. (2017). We trained for 80.000 generator iterations and 5 discriminator iterations per generator update for all of our networks. We also trained the AE for 80.000 iterations with ADAM using the default hyperparameters. The inversion of the generator networks was done using 10.000 ADAM steps using default hyperparameters. We found that the changes afterwards were negligible.
Generator Architecture (h, w, fout)
Discriminator Architecture (h, fout) SCon22D/LReeL,(to (32, 32, 256))
Densc/BN/ReLU (to (4, 4, 2048))
ResizeNN(to (8, 8, 2048))
Conv2D/LReLU (to (32, 32, 256))
CO (to (8, 8, 1024))
2e (16, 512)
ResizeNN(to (16, 16, 1024))
Conv2D/LReLU (to (16, 16,
Con/2D/BN/ReLU (to (16, 16, 512))
ReoLtto (8, 8, 1024))
ResizeNN(to (32, 32, 32,512)) 512))
Conv2D/LReLU (to (8, 8, 1024))
C (to (32, 32, 256))
SC (4, 4, 2048))
ResizeNN(to (64, 64, 256))
Conv2D/LLReIJU (to (4, 4, 2048))
Con/2DD/N//RLLUU (to (64, 64, 128))
Dense (to 1)
Conv2D/TanH (to (64, 64,3))
Table 1: Architecture of the Discriminator and Generator networks. Initialization is done using
Glorot & Bengio (0110)
PROOF OF CONCEPT ON MNIST
The initial experiments on MNIST validating the legitimacy of the approach are shown in Fig. 9 Instead of using visual quality metric to determine if the resulting images are similar enough, used the reconstructed training images of the GAN for training a LeNet Natwork[[eCun et al. (1998) to determine if the images are descriptive. The manifold of MNIST has been faithfully learned before by [Shmelkov et al. (2018). In contrast to their work, we show that this only works given large enough latent space and that the reconstruction error of the AE and the WGAN-GP converge for larger latent spaces.
0.5
WGAN-GP Tran
1.0
WGAN-GP Val
AE Train
AE Vel
0.8-
0.3
0.6
0.4
0.1
Reconstructed
6.0
0.2
38
dmiz)
dimiz)
(a)
(b)
Figure 9: Mnist GAN and AE reconstructions in (a). Visual quality metrics are not really applicable on MNIST data in (b) the reconstructed GAN images used to train LeNet to classify on MNIST and is compared to training on the true training set.
11

---

Under review as a conference paper at ICLR 2020
Normal Distributed Data
Reconstruction Test Set
Reconstruction Training Set
0.5
||=11a
Figure 10: The distribution of the reconstructions without projections.
DISTRIBUTION OF THE TRAINING SET IN THE GENERATOR MANIFOLD
In this section the main question if the found reconstructions likely appear in the generative model. For this reason the inverse optimization procedure is run without the projection onto the fea- sible set for dim(2) shown in Fig Fig The corresponding set of vectors, which is likely to appear in the training is shown in blue, while the training set reconstructions are shown in red and the test set reconstructions are shown in green. As is apparent therein, the overlap is virtually non-existant and this is the reason why we used a projected gradient method.
1000 on CIFAR-10. 10. The resulting latent vector norm of the reconstructions
ADDITIONAL GAN VS AE EXPERIMENTS
We also investigated the difference between the ability of GAN to reconstruct images of CIFAR-10 10 while trained on CelebA compared to AE. The results are shown in Fig. [ and as is already shown in Fig. 7 the GAN algorithm is better at reconstructing CIFAR-10 images than CelebA images. However, this is not the case for the AE.
And as final experiment we show visual results on CIFAR- images produced by an inverted GAN and by an AE in Fig. [2] As in the main paper we can the same behavior again. The AE produces initially blurry results and then sharp ones and the GAN produces sharp results of unrelated images and then produces images which are closer to the actual images.
12

---

Under review as a conference paper at ICLR 2020
0.20
AE
WGAN-GP
Cifar Error
0.10
0.05
0.00
Celeb Error
Figure 11: ROC Curve of training CelebA and testing the reconstruction enror of the NN on CIFAR-10 10 and CelebA. The desired behavior would be in the top left comer. The points correspond to setting with specific latent space. As the latent space decreases the error goes down. The points correspond to the target latent space size as shown in Fig.
233
230
1092
500
2000
dmmz))
chmtz))
(a) AE
(b) WGAN-GP
Figure 12: The first column are the actual images. As we increase the latent space of the autoencoder we can the image quality increasing. The same cannot be said about the GAN reconstructions. Those are of high quality even for low dimensional space, but correspond to other people.
13


Reviews:
[{"review_id": "SkxSd5_otS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Impact of the latent space on the ability of GANs to fit the distribution\n\nThis paper purports to study the behavior of latent variable generative models by examining how the dimensionality of the latent space affects the ability of said models to reconstruct samples from the dataset. The authors perform experiments with deterministic autoencoders and WGAN-GP on CIFAR, CelebA, and random noise, and measure MSE and FID as a function of the dimensionality of z.\n\nMy take: This paper does not offer any especially intriguing insights, and many of the conclusions the authors draw are, in my opinion, not supported by their experiments. The paper is confusingly written and hard to follow—throughout my read I struggled to determine what the authors meant, and it was not clear to me what this paper is supposed to contribute. The potential impact of this paper is very low, and I argue strongly in favor of rejection.\n\nNotes:\n\nMy most critical complaint is the central experiment set of the paper: measuring MSE and FID as a function of Dim-Z for two models. First of all, the authors assume that the reduction in MSE as a function of dim Z is indicative of increased memorization in the GAN models. I disagree that this is the case; since the GAN-based reconstruction is done via optimization it is unsurprising that increasing dim Z increases the reconstruction quality, as you are literally giving the model more degrees of freedom with which to fit the individual samples. This is glaringly evident in Figure 8, where increasing dim Z renders the model better able to reconstruct a sample of pure noise, which is almost certainly not in its normal output distribution, (or if it is, is in there with staggeringly low probability). The fact that the higher dim-z models are better able to reconstruct the noise supports the notion that it is merely the number of degrees of freedom that matter in these experiments, rather than what the model actually learns.\n\nSecond, it is important to note that FID can be easily gamed by memorization, and for an autoencoder (which has direct access to samples) with an increasingly large bottleneck it is unsurprising that increasing dim-Z tends to decrease the FID, and equally unsurprising that increasing the dim-Z for the GAN does not tend to improve results, since this does not really allow the model increased memorization capacity (not to mention the relationship between performance and dim-Z has been explored before in GAN papers).\n\nThird, the organization of the experimental section makes it very difficult to infer what the authors are trying to conclude from these experiments. The noise experiment is presented, but no insights or conclusions are drawn, other than (a) noting that the model has a harder time reconstructing the noise than training samples and (b) that lower dim models have a harder time reconstructing the noise, both of which are just restatements of the information presented in the figure rather than an actual insight or conclusion.\n\n-I’m not really sure what the experiment in section 3 is supposed to show. This experiment is poorly described and lacking details. First of all, what is the loss function used there? Is this the output of the discriminator or the MSE between the output and a target sample? How is z* found and what does it represent—is it just a randomly sampled latent, or is it the latent that corresponds to the z-value which minimizes some MSE loss for a target sample? If it’s the latter, why is this notation not introduced until section 4? If it’s a latent, why are you calling it a data point? Why are there no axes and no scales on these plots? How is it clear that there is an optimization path from z0 to z*; is that supposed to be inferred from z0 having a higher value than z* or appearing to be directly uphill from z*, because it’s not clear to me that that is the case in Figure 2a. In general I did not find this experiment to support the conclusions the authors draw.  \n\n-Figure 4: It is important to note that FID can be trivially gamed by memorizing the dataset, and an autoencoder is much more well-suited to memorizing the dataset as it has direct access to samples (whereas a GAN must get them through the filter of the discriminator). Authors should test interpolation or hold-out likelihood for the  autoencoder, these models are not directly comparable in this manner.\n\n-The presentation of this paper is, in general, all over the place. The authors should focus on writing such that each point follows the next, building progressively towards their results and insights, and making it easy for a reader to follow their train of thought.\n\n“In this work, we show that by reducing the problem to a compression task, we can give a lower bound on the required capacity and latent space dimensionality of the generator network for the distribution estimation task.” At what point is this lower bound (either in terms of model capacity or latent space dimensionality) specified in the paper? Is Figure 3 supposed to be this lower bound, because to me it only indicates that the autoencoder tends to have a lower MSE, not that it conclusively lower bounds the memorization capacity of the GAN. Wouldn't a method like GLO which directly optimizes for memorization be a better lower bound for this, anyhow?\n\n“We rely on the assumption, that less capacity is needed to reconstruct the training set, than to reconstruct the entire distribution” What does this phrase mean? Are the authors referring to the entire distribution of natural images, of which the training set is assumed to be a subset? Or do they mean the output distribution of the generator? This was not clear to me.\n\n\nMinor:\n\n-“ style transfer by Karras et al. (2018),”, and “anomaly detection (Shocher 2018).” StyleGAN is not a style transfer paper, and InGAN  is not about anomaly detection. Please do not incorrectly summarize papers.\n\n-“Trained GAN newtworks” While amusing, this is a typo. Please thoroughly read your paper and correct all typos and grammatical mistakes, like “combiniation.”\n\n-“…that an accurate reconstruction of the generator manifold is possible works using first order methods”  The word “works” seems to be out of place here. Again, please thoroughly proofread your paper.\n\n-The legend in Figure 2 has a white background, making the white x corresponding to z0 invisible. Please fix this, and add appropriate axes to this plot.\n\n-Figure 7 and 8 may in fact have error bars, but they are not described (are they 1 std or another interval?) or referenced, and in Figure 8 (if these are error bars) they are nearly invisible. \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SkxN3oeEqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper explores the influence of the dimensionality of the latent space to the quality of the learned distributions for autoencoders (AE) and GANs (more precisely Wasserstein GANs). In particular, the paper looks at the ability of the learned AE or GAN to reconstruct the training images, the visual quality of the images as the dimension of the latent space increases, and the ability to reconstruct images not in the training set (structured in some ways). \n\nEvaluation: While the general flavor of question the paper studies is undoubtedly interesting, I found the paper severely lacking both in terms of the quality of writing (in particular, I was at confused about the goal of various sections/experiments), as well as the significance of the results the authors observe (and how they are reported -- I found them to be oversold). \n\nRegarding the quality of results: \n\n* The paper primarily talks about the ability of AEs and GANs to *reconstruct* images, either in the training set, or in the test set, or in some different dataset altogether (e.g. shifted images, different image dataset). This is a problematic thing on multiple levels: first, the goal of a GAN or AE is to fit a distribution -- merely having a data point in its domain says nothing about the *probability* of that point; second, the way these \"spans\" are tested is via running a gradient descent search for a pre-image for the data point. The authors never comment or explore whether the problem may *not* be that these data points are not in the image of the GAN, but rather that the optimization procedure doesn't succeed. (And indeed, increasing the dimensionality of the latent space may act as \"overparametrization\" for this gradient descent procedure, making it more likely to succeed.) \n\nFinally, there are some fairly arbitrary choices in the entire experimental setup: why WGANs vs another architecture -- are the GAN results sensitive to architecture choices? why AEs and not VAEs (and with what variational posterior) -- how sensitive are the observations here to choosing the most vanilla variant of autoencoders? These are all questions that invariably linger after reading the paper. \n\nRegarding the quality of writing: \n\n* There are various sloppy sentences in crucial parts of the paper. I will only list a few:   \n-- \"Once a suitable AE for a dataset is found, the decoder part of is used as the generative model of choice.\" -- this seems to suggest a semi-synthetic setup where an AE is trained to use as a generator of a data set for which a GAN is fit. I never saw this setup in Section 5 -- although this would be a good way to test \"relative\" representational power of GANs and AEs. \n-- \"In principle, getting from an AE to a GAN is just a rearrangement of the NNs\" in Section 5.1 -- I wasn't sure what this is supposed to mean, and this is a critical part of that section. \n-- \"The AE network acts as a lower bound for the GAN algorithm, therefore validating our intuition that the AE complexity lower-bounds the GAN\" in Section 5.1 -- also very sloppy, and I'm not sure what it means -- I guess the authors mean the reconstruction performance of AE is a lower bound on the GAN reconstruction. Not sure what this has to do with \"complexity\". \n\n* Various sections are meandering, and I wasn't sure what the goal is. Just a few examples: section 3 spends a lot of time talking about known theoretical results wrt. to invertibility of random-like neural nets. It wasn't clear to me how this relates to the results in Section 5, especially since the authors never leverage/talk about these theory results again. (Instead, they study empirical invertibility via gradient-descent based procedures.) Similarly, interpolating by polynomials is talked about in (2), seemingly without any point. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1ecMH32YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work performs a systematic empirical study of how the latent space design in a GAN impacts the generated distribution. The paper is well-written and was easy to read. Also, I find this to be an interesting and promising direction of research.\n\nThe convergence proof in Goodfellow (2014) assumes that the data distribution has a density, and essentially states that the JS-divergence is zero if and only if the two distributions are the same. In practice, the data distribution is discrete, while the latent distribution has a probability density function. It is not possible to transform a density into a discrete distribution by a continuous map and neural networks are always continuous by construction. In theory, as training progresses, more and more latent mass will be pushed on the discrete samples and no minimizer exists (unless the function space of generator is constrained or the real distribution is smoothed out a bit). \n\nSince it is not possible to assess whether the GAN training has converged due the nonconvexity of the energy and non-existence of a global optimizer, the empirically observed results might be very specific to the chosen optimization procedure, stopping criterion, dataset, hyper-parameters, initialization, network architectures, etc etc.  It is a challenge to study the choice of latent space in a somewhat \"isolated\" way. These issues should be discussed in the paper and the reader should be made aware of such problems.\n\nAnother point, could it be, that by increasing the dimension of the latent space, one makes it easier for the nonconvex optimization in (5) to converge to \"unlikely but realistic looking samples\"? I think this is not too far-fetched, as increasing the dimension of an optimization problem often makes local optimization less likely to get stuck at local optima. Also it might not be the best idea to optimize (5) with Adam since it is not a stochastic optimization problem and there are provably convergent solvers out there for this problem class. \n\nSince it is possible to evaluate the likelihood of the optimized reconstructions that are nearby the data points, one could check whether this is indeed the case. While constrained not to be too unlikely, I wonder whether the likelihood increases or decreases with the dimensionality of the latent space and this would make an interesting plot. \n\nUnfortunately, I did not understand the connections to auto-encoders, as they might optimize a fundamentally different criterion than GANs. In particular \"In principle, getting from an AE to a GAN is just a rearrangement of the NNs. \" is unclear to me. \n\nAlso, what is meant by lower-bound? Is the claim that the reconstruction error in an auto encoder will be lower, than if one optimizes the latent code in a GAN to reconstruct the input? Figure 3 seems to support this hypothesis, but I don't have an intuition why this should be true and have some doubts. A mathematical proof seems out of reach. \n\nI have trouble to understand the \"intuition that the AE complexity lower-bounds the GAN complexity.\" Before reading this paper, my intuition was the opposite: If the generator distribution covers the real distribution, the reconstruction error for GAN is zero. Intuitively, it seems a much easier task to somehow cover a distribution than to minimize an average reconstruction error. \n\nThe connection of WGANs to the L2 reconstruction loss in the auto-encoder is very hand-wavy. It is still an open question whether WGANs actually have anything to do with the Wasserstein distance. People working in optimal transport doubt this, due to huge amount of approximations going on. \n\nAt this point I'm reluctant to recommend acceptance, as the paper tries to connect things, which for me are quite disconnected and the evaluations of reconstruction error, etc. might depend in intricate ways on the nonconvex optimization procedures.\n\nMinor suggestions, typos, etc (no influence on my rating):\n\n- What is the \"generated manifold\" that is talked about in the introduction, contributions and throughout the paper? To me, it is not directly clear that the support of the transformed distribution will be a manifold (especially if G is non-injective). Anyway, the manifold structure is nowhere exploited in the paper, so I suggest to call it \"transformed latent distribution\".\n\n- Had to pause a little bit to understand Eq. 2 (simple polynomial interpolation). It is unnecessary to show the explicit form, as I'm sure no one doubts the existence of a smooth curve interpolating a finite set of points in R^d.  \n\n- Equations should always include punctuation marks. \n\n- Eq. 5: dim --> \\text{dim} and s.t. --> \\text{s. t.}\n\n- Fig 3b: the red curve is missing or hidden behind another curve.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have pointed out several major deficiencies of the paper, which the authors decided not to address.
Decision: Reject