Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: 1. Regarding the masking technique, the paper is vague as to what the gradient step is minimizing. Could you elaborate?

2. Does the IAF loss function include a KL-divergence term between the approximate posterior distribution and the prior distribution like is done for VAEs?
review_title: Two questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents two main contributions:

(1) A novel model visualization and photo manipulation technique that allows to transform an image using a paintbrush, much like in an image editing software.

(2) A hybridization of GANs and VAEs called Introspective Adversarial Network.

The main problem I have with the paper is that it feels very much like two papers in one with a very loose story tying the two together.

On one hand, the neural photo editing technique is presented in sufficient detail to be reproducible and it is shown to be effective. I personally find the idea exciting, but in order for it to be of interest to the ICLR community I think more emphasis should be put on what insights such a technique allows to gain on trained models.

On the other hand, the IAF model is introduced, along with multiple network architecture modifications for improving its performance. One criticism that I have regarding the presentation is that it makes it hard to assign credit to individual ideas when they are presented in a "list of things to make it work" fashion. I would like to see more empirical results in that direction to help clear up things.

Overall I think the paper proposes interesting ideas, but given its lack of focus on a single, cohesive story I think it is not yet ready for publication.

UPDATE: The rating has been updated to a 6 following the authors' reply.
review_title: The paper lacks focus.

final_score: rating: 6: Marginally above acceptance threshold
review_content: After rebuttal:

I think the presentation improved in the revised version (although still quite cluttered and confusing), and new quantitative results look quite convincing. Therefore I raise my rating. Still, the paper could use polishing. If written in a better way, it would be a definite accept in my opinion.

---------
Initial review:

The paper presents a tool for exploring latent spaces of generative models, and "introspective adversarial network" model - a new hybrid of a generative adversarial network (GAN) and a variational autoencoder (VAE). On the plus side, the presented tool is interesting and may be useful for analysis of generative models, and the proposed architecture seems to perform well. On the downside, experimental evaluation does not allow for confident conclusions, and a recent closely related work by Zhu et al. [1] is not discussed in enough detail. Overall, I am in the borderline mode, and may change my opinion depending on how the discussion phase goes. 

Detailed comments:

1) The presented model combines elements of a large number of existing techniques: GAN, VAE, VAE/GAN (Lamb et al. 2016), inverse autoregressive flow (IAF), PixelRNN, ResNet, dilated convolutions. In addition, the authors propose new modifications: orthogonal regularization (inspired by Saxe et al.), ternary discriminator in a GAN. This makes the overall architecture complicated. An extensive ablation study could allow to judge about the effect of different components, but the ablation study presented in the paper is somewhat restricted. What do we learn from the proposed architecture? Can other researchers gain any new insights? What is important, what is not? Answering these question would significantly raise the potential impact of this paper.

2) Related to the previous point, proper analysis requires adequate measures of performance. Qualitative results are nice, but with the current surge of interest in generative models it gets very difficult to rely on qualitative evaluations: many methods produce visually similar results, and unless there is an obvious large jump in the quality of the produced images, it is unclear how to compare these. I do appreciate the effort authors have already put into evaluating the model: especially the keypoint error is interesting. Unfortunately, none of the presented measures evaluates visual quality of the images. A user study would be useful - I realize it is additional effort, but what is so restrictively difficult about it? Perhaps not with AMT, but with some fellow researchers/students.

3) Work [1] looks very related to the proposed visualization tool and deserves a more thorough discussion than a single sentence in the related work section. The paper by Zhu et al. appeared on arxiv more than 1,5 months before the ICLR deadline, and, more importantly, it has been published at ECCV before the ICLR deadline. This is unfortunate for the authors, but I think this makes the paper count as prior work, not concurrent. In the end this is up to ACs and PCs to decide. Anyway, I strongly suggest the authors to add a detailed discussion of differences of the two approaches, their capabilities, strengths and weaknesses. The authors could also try to directly compare to the approach of Zhu et al. or explain why it is impossible.

4) May be a good idea to extend Appendix A with more approaches (VAE and DCGAN are not state of the art, are they? why not show at least VAE/GAN?) and more datasets. If this is not possible, please explain why. Samples of faces from IAN do look very impressive, but a fair comparison with SOTA would strengthen your point. By the way, I assume the samples are random, not cherry-picked? Please mention it in the paper. 

5) The analysis capabilities of the proposed tool are not fully explored. What does it teach us about generative models? Does it work on non-face datasets? Overall, it seems that since the paper includes two largely disjoint contributions (a tool and a generative model), none of the two gets analyzed in depth, which makes the paper look somewhat incomplete.  

Small remarks:

1) Why not include 8% result of IAN on SVHN into the table? It is easy to miss otherwise. From the table it is absolutely unclear that some of the results are not comparable. The table should be more self-explanatory.

[1] Zhu et al., "Generative Visual Manipulation on the Natural Image Manifold", ECCV 2016, https://arxiv.org/pdf/1609.03552v2.pdf

review_title: Final review

final_score: rating: 5: Marginally below acceptance threshold
review_content: UPDATE: The rewritten paper is more focused and precise than the previous version. The ablation studies and improved evaluation of the IAN model help to the make clear the relative contributions of the proposed MDC and orthogonal regularization. Though the paper is much improved, in my opinion there is still too much emphasis on the photo-editing interface. In addition, the MDC blocks are used in the generator of the model but their efficacy is measured via discriminative experiments. All-in-all I am updating my score to a 5.

==========

This paper proposes a hybridization of a VAE and a GAN whereby the generator both generates random samples and produces reconstructions of the real data, and the discriminator attempts to classify each true data point, sample, and reconstruction as being real, fake, or reconstructed. It also proposes a user-facing interface with an interactive image editing algorithm along with various modifications to standard generative modeling architectures.

Pros:
+ The IAN model itself is interesting as standard GAN-based approaches do not simultaneously train an autoencoder.

Cons:
- The writing is unclear at times and the mathematical formulation of the IAN is not very precise.
- Many different ideas are proposed in the paper without sufficient empirical validation to characterize their individual contributions.
- The photo editing interface, though interesting, is probably better suited for a conference with more of an HCI focus.

* Section 2: The gradient descent step procedure seems to be quite similar to the approach proposed by [2]. More elaboration on the differences would be helpful.
* Section 3: It is unclear whether the discriminator has three binary outputs or if there is a softmax over the three possible labels. The paper does not mention whether L_G and L_D are minimized or maximized. Presumably they are both minimized, but in that case it is counter-intuitive that the generator attempts to decrease the probability that the discriminator assigns the "real" label to the generated samples and reconstructions. In addition, in the minimization scenario L_D maximizes the probability of the correct labels being assigned to X_gen and \hat{X} but minimizes the probability of the "real" label being assigned to X.
* Section 3.2: It is odd that not training MADE leads to better performance, as training MADE should lead to a better variational approximation to the true posterior. More exploration seems warranted here.
* Section 3.3.2: One drawback of autoregressive approaches is that sampling is slow. How do you reconcile this with its use in an interactive application, where speed is important?
* Figure 7: The Inception score is typically expressed as an exponentiated KL-divergence. It is odd that the scores are being presented here as percents.
* Section 4.1: The Inception score's direct application to non-natural images is indeed problematic. One potential workaround is to compute exponentiated KL for a discriminative net trained specifically for the dataset, e.g. to predict binary attributes on CelebA. 

Overall, the paper attempts to simultaneously do too many things. It could be made much stronger by focusing on the primary contribution, the IAN, and performing a comparison against similar approaches such as [1]. The other techniques, such as IAF with randomized MADE, MDC blocks, and orthogonal regularization are potentially interesting in their own right but the current results are not conclusive as to their specific benefits.

[1] Larsen, Anders Boesen Lindbo, Søren Kaae Sønderby, and Ole Winther. "Autoencoding beyond pixels using a learned similarity metric." arXiv preprint arXiv:1512.09300 (2015).
[2] J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros, “Generative Visual Manipulation on the Natural Image Manifold,” ECCV 2016.
review_title: Final Review

final_score: rating: null
review_content: Hello,

A couple of questions:

1. How does this paper relate to Zhu et al. "Generative Visual Manipulation on the Natural Image Manifold", ECCV 2016 ?

2. Have you performed any quantitative evaluations except for semi-supervised learning on SVHN ? I know evaluating generative models is difficult, but there are some methods used in the literature https://arxiv.org/abs/1606.03498 https://arxiv.org/pdf/1610.09585v1.pdf . Would these not be applicable?

3. In your qualitative results, could you compare to established baselines and show more results? In many figures you show just two images, and it is really difficult to judge from those.

Thanks.
review_title: pre-review questions

final_score: rating: null
review_content: 
1. This may be a typo, but D(X_{gen}) does not appear to enter into the discriminator loss in the equation at the end of page four (please number the equations so they can be referred to more easily).
2. In the expression for L_G, how is the tradeoff between the pixel-wise reconstruction loss and the feature-wise reconstruction loss handled?
3. Have you considered ways to quantitatively evaluate alignment of facial landmarks in reconstructions, or preservation of facial features?
4. In Figure 7, it is difficult to assess the effects of the various models with a single reconstruction per example. One possible way to explore this would be to generate multiple reconstructions by sampling from \hat{Z}.
review_title: Pre-review questions

Meta-review: Here is a summary of strengths and weaknesses as per the reviews:
 
 Strengths
 Work/application is exciting (R3)
 Enough detail for reproducibility (R3)
 May provide a useful analysis tool for generative models (R1)
 
 Weaknesses
 Clarity of the IAN model - presentation is scattered and could benefit from empirical analysis to tease out which parts are most important (R3,R2,R1); AC comments that the paper was revised in this regard and R3 was satisfied, updating their score
 Lack of focus: is it the visualization/photo manipulation technique, or is it the generative model? (R3, R2)
 Writing could use improvement (R2)
 Mathematical formulation of IAN not precise (R2)
 
 The authors provided a considerable overhaul of the paper, re-organizing/re-focusing it in response to the reviews and adding additional experiments. 
 
 This, in turn, resulted in R1, R2 and R3 upgrading their scores. The paper is more clearly in accept territory, in the ACÕs opinion. The AC recommends acceptance as a poster.
Decision: Accept (Poster)