Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is well written and easy to follow. I find the results very interesting. In particular the paper shows that many properties of GAN (or generative) models (e.g. interpolation, feature arithmetic) are a in great deal result of the inductive bias of deep CNN’s and can be obtained with simple reconstruction losses. 

The results on CelebA seem quite remarkable for training examples (e.g. interpolation). Samples are quite good but inferior to GANs, but still impressive for the simplicity of the model. The results on SUN are a bit underwhelming, but still deliver the point reasonably well in my view. Naturally, the paper would make a much stronger claim showing good results on different datasets. 

The authors mentioned that the current method can recover all the solutions that could be found by an autoencoder and reach some others. It would be very interesting to empirically explore this statement. Specifically, my intuition is that if we train a traditional autoencoder (with normalization of the latent space to match this setting) and compute the corresponding z vectors for each element in the dataset, the loss function (1) would be lower than that achieved with the proposed model. If that is true, the way of solving the problem is helping find a solution that prevents overfitting. 

Following with the previous point, the authors mention that different initializations were used for the z vectors in the case of CelebA and LSUN. Does this lead to significantly different results? What would happen if the z values were initialized say with the representations learned by a fully trained deterministic autoencoder (with the normalization as in this work)? It would be good to report and discuss these alternatives in terms of loss function and results (e.g. quality of the samples). 

It seems natural to include VAE baselines (using both of the losses in this work). Also, recent works have used ‘perceptual losses’, for instance for building VAE’s capable of generating sharper images:

Lamb, A., et al (2016). Discriminative regularization for generative models. arXiv preprint arXiv:1602.03220.

It would be good to compare these results with those presented in this work. One could argue that VAE’s are also mainly trained via a regularized reconstruction loss. Conversely, the proposed method can be thought as a form of autoencoder. The encoder could be thought to be implicitly defined by the optimization procedure used to recover the latent vectors in GAN's. Using explicit variables for each image would be a way of solving the optimization problem.

It would be informative to also shows reconstruction and interpolation results for a set of ‘held out’ images. Where the z values would be found as with GANs. This would test the coverage of the method and might be a way of making the comparison with GANs more relevant. 

The works:

Nguyen, Anh, et al. "Synthesizing the preferred inputs for neurons in neural networks via deep generator networks." Advances in Neural Information Processing Systems. 2016.

Han, Tian, et al. "Alternating Back-Propagation for Generator Network." AAAI. 2017.

Seems very related.

review_title: Official review

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The authors observe that the success of GANs can be attributed to two factors; leveraging the inductive bias of deep CNNs and the adversarial training protocol. In order to disentangle the factors of success, and they propose to eliminate the adversarial training protocol while maintaining the first factor. The proposed Generative Latent Optimization (GLO) model maps a learnable noise vector to the real images of the dataset by minimizing a reconstruction loss. The experiments are conducted on CelebA and LSUN-Bedroom datasets. 

Strengths: 
The paper is well written and the topic is relevant for the community.
The notations are clear, as far as I can tell, there are no technical errors.
The design choices are well motivated in Chapter 2 which makes the main idea easy to grasp. 
The image reconstruction results are good. 
The experiments are conducted on two challenging datasets, i.e. CelebA and LSUN-Bedroom.

Weaknesses:
A relevant model is Generative Moment Matching Network (GMMN) which can also be thought of as a “discriminator-less GAN”. However, the paper does not contrast GLO with GMMN either in the conceptual level or experimentally. 

Another relevant model is Variational Autoencoders (VAE) which also learns the data distribution through a learnable latent representation by minimizing a reconstruction loss. The paper would be more convincing if it provided a comparison with VAE.

In general, having no comparisons with other models proposed in the literature as improvements over GAN such as ACGAN, InfoGAN, WGAN weakens the experimental section.

The evaluation protocol is quite weak: CelebA images are 128x128 while LSUN images are 64x64. Especially since it is a common practice nowadays to generate much higher dimensional images, i.e. 256x256, the results presented in this paper appear weak. 

Although the reconstruction examples (Figure 2 and 3) are good, the image generation results (Figure 4 and 5) are worse than GAN, i.e. the 3rd images in the 2nd row in Figure 4 for instance has unrealistic artifacts, the entire Figure 5 results are quite boxy and unrealistic. The authors mention in Section 3.3.2 that they leave the careful modeling of Z to future work, however the paper is quite incomplete without this.

In Section 3.3.4, the authors claim that the latent space that GLO learns is interpretable. For example, smiling seems correlated with the hair color in Figure 6. This is a strong claim based on one example, moreover the evidence of this claim is not as obvious (based on the figure) to the reader. Moreover, in Figure 8, the authors claim that the principal components of the GLO latent space is interpretable. However it is not clear from this figure what each eigenvector generates. The authors’ observations on Figure 8 and 9 are not clearly visible through manual inspection.  

Finally, as a minor note, the paper has some vague statements such as
“A linear interpolation in the noise space will generate a smooth interpolation of visually-appealing images”
“Several works attempt at recovering the latent representation of an image with respect to a generator.”
Therefore, a careful proofreading would improve the exposition. 
review_title: OPTIMIZING THE LATENT SPACE OF GENERATIVE NETWORKS

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors propose a new architecture for generative neural networks. Rather than the typical adversarial training procedure used to train a generator and a discriminator, the authors train a generator only. To ensure that noise vectors get mapped to images from the target distribution, the generator is trained to map noise vectors to the set of training images as closely as possible. Both the parameters of the generator and the noise vectors themselves are optimized during training. 

Overall, I think this paper is useful. The images generated by the model are not (qualitatively and in my opinion) as high quality as extremely recent work on GANs, but do appear to be better than those produced by DCGANs. More importantly than the images produced, however, is the novel training procedure. For all of their positive attributes, the adversarial training procedure for GANs is well known to be fairly difficult to deal with. As a result, the insight that if a mapping from noise vectors to training images is learned directly, other noise images still result in natural images is interesting.

However, I do have a few questions for the authors, mostly centered around the choice of noise vectors.

In the paper, you mention that you "initialize the z by either sampling them from a Gaussian distribution or by taking the whitened PCA of the raw image pixels." What does this mean? Do you sample them from a Gaussian on some tasks, and use PCA on others? Is it fair to assume from this that the initialization of z during training matters? If so, why?

After training, you mention that you fit a full Gaussian to the noise vectors learned during training and sample from this to generate new images. I would be interested in seeing some study of the noise vectors learned during training. Are they multimodal, or is a unimodal distribution indeed sufficient? Does a Gaussian do a good job (in terms of likelihood) of fitting the noise vectors, or would some other model (even something like kernel density estimation) allow for higher probability noise vectors (and therefore potentially higher quality images) to be drawn? Does the choice of distribution even matter, or do you think uniform random vectors from the space would produce acceptable images?
review_title: This paper is a potentially interesting alternative training procedure to GANs.

Meta-review: This paper attempts to decouple two factors underlying the success of GANs: the inductive bias of deep CNNs and adversarial training. It shows that, surprisingly, the second factor is not essential. R1 thought that comparisons to Generative Moment Matching Networks and Variational Autoencoders should be provided (note: this was added to a revised version of the paper). They also pointed out that the paper lacked comparisons to newer flavors of GANs. While R1 pointed out that the use of 128x128 and 64x64 images was weak, I tend to disagree as this is still common for many GAN papers. R2 was neutral to positive about the paper and thought that most importantly, the training procedure was novel. R3 also gave a neutral to positive review, claiming the paper was easy to follow and interesting. Like R1, R3 thought that a stronger claim could be made by using different datasets. In the rebuttal, the authors argued that the main point was not in proposing a state-of-the-art generative model of images but to provide more an introspection on the success of GANs. Overall, I found the work interesting but felt that the paper could go through one more review/revision cycle. In particular, it was very long. Without a champion, this paper did not make the cut.
Decision: Reject