Title: Neural Photo Editing with Introspective Adversarial Networks
Authors: Andrew Brock
Theodore Lim
J.M. Ritchie
Nick Weston
Venue: ICLR 2017
Keywords: Computer vision
Unsupervised Learning
Applications

Abstract: The increasingly photorealistic sample quality of generative image models suggests their feasibility in applications beyond image generation. We present the Neural Photo Editor, an interface that leverages the power of generative neural networks to make large, semantically coherent changes to existing images. To tackle the challenge of achieving accurate reconstructions without loss of feature quality, we introduce the Introspective Adversarial Network,  
a novel hybridization of the VAE and GAN. Our model efficiently captures long-range dependencies through use of a computational block based on weight-shared dilated convolutions, and improves generalization performance with Orthogonal Regularization, a novel weight regularization method. We validate our contributions on CelebA, SVHN, and CIFAR-100, and produce samples and reconstructions with high visual fidelity.

TL;DR: An interface for editing photos using generative image models.

Review discussion:
1.
Regarding the masking technique, the paper is vague as to what the gradient step is minimizing. Could you elaborate?
2. Does the IAF loss function include a KL-divergence term between the approximate posterior distribution and the prior distribution like is done for VAEs?
Hi,
Thanks for the questions!
1. The gradient descent step is minimizing the local L2 difference between the output image and the requested color, evaluated at the current paintbrush location.
2. The IAF loss does not include a KL divergence term (as the MADE is not trained) but the encoder subnetwork loss does have a KL divergence between the approximate posterior q(\hat{Z}|X) and the isotropic gaussian prior p(Z).
We have updated the appropriate sections paper to clarify both of these issues.
Thanks,
Andy
Hello,
A couple of questions:
1. How does this paper relate to Zhu et al. "Generative Visual Manipulation on the Natural Image Manifold", ECCV 2016 ?
2. Have you performed any quantitative evaluations except for semi-supervised learning on SVHN ? I know evaluating generative models is difficult, but there are some methods used in the literature https://arxiv.org/abs/1606.03498 https://arxiv.org/pdf/1610.09585v1.pdf . Would these not be applicable?
3. In your qualitative results, could you compare to established baselines and show more results? In many figures you show just two images, and it is really difficult to judge from those.
Thanks.
Hi,
Thanks for the questions!
1. Zhu et al is a contemporaneous and independently developed work (released within a few days of ours) with several key differences:
-They train an inference network separately (rather than jointly) and use its predicted latent values as initial values which they then iterate to optimize the L2 reconstruction loss, while we take a one-shot reconstruction using the inferred latents.
-User inputs are taken as constraints which are used to solve for latent changes via multiple optimization steps after the user provides input, while we update our images in real time as the user paints.
-Changes to output images are transferred to original images via a color and flow estimation, as opposed to the pixel-wise masking which we use to directly transfer edits between images. (They specifically cite the difficulty of directly transferring pixel-wise edits, which we also found to be true, hence the introduction of the masked interpolation).
-The domain focus of their work also differs, in that they focus more on editing clothing articles and landscape images, while we focus on faces.
We’ve updated the paper to include a reference to this work, as well as to the Neural Doodle (interactive style transfer) work of Champanard.
2. We acknowledge that a lack of more quantitative evaluations is frustrating;
as to the mentioned methods:
-We lack the resources to engage human annotators for this particular task.
-The Inception score proposed in Improved-GAN requires a pre-trained model from the same domain, which in our opinion only really provides insight into our ImageNet samples, a minor focus of our work. We tested our four CelebA models compared in Figure 7 using the Inception score evaluator provided in the Improved-GAN source code and noticed relatively little difference between them (as compared to the difference in Inception scores reported by Salimans et al for CIFAR samples in Improved-GAN).
We suspect that this is because the model used for evaluation is trained on Imagenet, rather than on a faces dataset, but we have included this score (along with L2 and L1 reconstruction scores on the CelebA validation set) in Figure 7. Evaluating the inception score using an appropriately trained classifier might provide more insight, but the only public CelebA classifier we’re aware of is that of Lamb et al (https://github.com/vdumoulin/discgen), which is only provided as an architecture (no weights).
-The Inception Accuracy proposed in Auxiliary Classifier GANs is not applicable as it assumes class-conditional generation, whereas our models are trained in a wholly unsupervised setting (i.e. without attribute or class labels).
3. We initially omitted such comparisons for brevity, but the consensus appears to be that this would be a worthwhile addition, so we’ve updated the paper to include an appendix with a comparison of samples between a VAE, DCGAN, and IAN with identical architectures trained on celebA. If the reviewers believe additional comparisons would be pertinent (i.e. to ALI/BiGAN or DiscGen) we can include those as well.
We considered adding reconstruction comparisons to ALI in particular, but opted not to as (a) the images provided in the ALI paper are compressed and (b) we felt that a fair comparison would involve comparing reconstructions to the same set of original images, but again lacked access to publicly available weights for their model, and were unable to determine exactly which indices from the validation or test set were selected for reconstruction.
Thanks,
Andy
1. This may be a typo, but D(X_{gen}) does not appear to enter into the discriminator loss in the equation at the end of page four (please number the equations so they can be referred to more easily).
2. In the expression for L_G, how is the tradeoff between the pixel-wise reconstruction loss and the feature-wise reconstruction loss handled?
3. Have you considered ways to quantitatively evaluate alignment of facial landmarks in reconstructions, or preservation of facial features?
4. In Figure 7, it is difficult to assess the effects of the various models with a single reconstruction per example. One possible way to explore this would be to generate multiple reconstructions by sampling from \hat{Z}.
Hi,
Thanks for the questions!
1. We initially included X_{gen} as G(Z_{rand}) in these equations as we thought this would be more clear. Evidently this is not the case, (and is inconsistent with our other notation of X and \hat{X}) so we have replaced G(Z_{rand}) with X_{gen} in the appropriate locations and numbered our equations.
2. We include a scalar weighting term (gamma) on the L1 reconstruction loss, which weights the importance of pixel-wise accuracy relative to the other losses. In our code we provide the options to weight each of the losses, but in practice we only vary gamma. For all experiments, gamma is kept at 3.0. We have updated the paper to include this information.
3. We previously relied on L1 and L2 loss as the primary quantitative metrics of reconstruction accuracy, but are currently attempting to compare our models using the recently published fiducial detection system of Ranjan et al (https://arxiv.org/abs/1611.00851) provided at umdfaces.io.
4. We kept these figures small for the sake of brevity, but have augmented Figure 7 with quantitative evaluations (L1 and L2 reconstruction loss and the Inception score of Salimans. et al) and included an appendix comparing samples from our models with those from VAE and DCGAN.
Thanks,
Andy
Hello,
In response to reviewer questions, we have evaluated and reported the fiducial keypoint reconstruction accuracy using the system of Ranjan et al (https://arxiv.org/abs/1611.00851). Figure 7 is now updated to include this information.
Best,
Andy
After rebuttal:
I think the presentation improved in the revised version (although still quite cluttered and confusing), and new quantitative results look quite convincing.
Therefore I raise my rating.
Still, the paper could use polishing. If written in a better way, it would be a definite accept in my opinion.
Initial review:
The paper presents a tool for exploring latent spaces of generative models, and "introspective adversarial network" model - a new hybrid of a generative adversarial network (GAN) and a variational autoencoder (VAE). On the plus side, the presented tool is interesting and may be useful for analysis of generative models, and the proposed architecture seems to perform well. On the downside, experimental evaluation does not allow for confident conclusions, and a recent closely related work by Zhu et al. [1] is not discussed in enough detail.
Overall, I am in the borderline mode, and may change my opinion depending on how the discussion phase goes.
Detailed comments:
1) The presented model combines elements of a large number of existing techniques: GAN, VAE, VAE/GAN (Lamb et al. 2016), inverse autoregressive flow (IAF), PixelRNN, ResNet, dilated convolutions.
In addition, the authors propose new modifications: orthogonal regularization (inspired by Saxe et al.), ternary discriminator in a GAN. This makes the overall architecture complicated. An extensive ablation study could allow to judge about the effect of different components, but the ablation study presented in the paper is somewhat restricted. What do we learn from the proposed architecture? Can other researchers gain any new insights? What is important, what is not? Answering these question would significantly raise the potential impact of this paper.
2) Related to the previous point, proper analysis requires adequate measures of performance. Qualitative results are nice, but with the current surge of interest in generative models it gets very difficult to rely on qualitative evaluations: many methods produce visually similar results, and unless there is an obvious large jump in the quality of the produced images, it is unclear how to compare these. I do appreciate the effort authors have already put into evaluating the model: especially the keypoint error is interesting. Unfortunately, none of the presented measures evaluates visual quality of the images. A user study would be useful - I realize it is additional effort, but what is so restrictively difficult about it? Perhaps not with AMT, but with some fellow researchers/students.
3) Work [1] looks very related to the proposed visualization tool and deserves a more thorough discussion than a single sentence in the related work section. The paper by Zhu et al. appeared on arxiv more than 1,5 months before the ICLR deadline, and, more importantly, it has been published at ECCV before the ICLR deadline. This is unfortunate for the authors, but I think this makes the paper count as prior work, not concurrent.
In the end this is up to ACs and PCs to decide. Anyway, I strongly suggest the authors to add a detailed discussion of differences of the two approaches, their capabilities, strengths and weaknesses. The authors could also try to directly compare to the approach of Zhu et al. or explain why it is impossible.
4) May be a good idea to extend Appendix A with more approaches (VAE and DCGAN are not state of the art, are they? why not show at least VAE/GAN?) and more datasets. If this is not possible, please explain why. Samples of faces from IAN do look very impressive, but a fair comparison with SOTA would strengthen your point. By the way, I assume the samples are random, not cherry-picked? Please mention it in the paper.
5) The analysis capabilities of the proposed tool are not fully explored. What does it teach us about generative models? Does it work on non-face datasets?
Overall, it seems that since the paper includes two largely disjoint contributions (a tool and a generative model), none of the two gets analyzed in depth, which makes the paper look somewhat incomplete.
Small remarks:
1) Why not include 8% result of IAN on SVHN into the table? It is easy to miss otherwise. From the table it is absolutely unclear that some of the results are not comparable. The table should be more self-explanatory.
[1] Zhu et al., "Generative Visual Manipulation on the Natural Image Manifold", ECCV 2016, https://arxiv.org/pdf/1609.03552v2.pdf
Hi,
Thanks for the review! We have thoroughly revised the paper to take into account reviewer comments.
In particular, we have changed the focus of the paper to the specific design design challenges towards which the IAN is focused, and include the Neural Photo Editor as a proof-of-concept demonstrating the viability of our proposed method, in the hopes of mollifying the disjoint focus previously noted. We have also removed our section on IAF with randomized MADE and RGB/Beta blocks, and run a set of in-depth ablation studies on CIFAR-100 and CelebA to evaluate the relative usefulness of Orthogonal Regularization, our Multiscale Dilated Convolution blocks, and the Ternary Adversarial Loss. Due to time constraints, we were unable to re-run our 128x128 ImageNet experiments (which made use of IAF and RGB/Beta blocks), so we have withdrawn our ImageNet samples and reconstructions. We are currently training networks on tiny-Imagenet and CIFAR, and will hopefully update the paper when those have finished training.
1) We have removed our two least important contributions (IAF with randomized MADE and RGB/Beta blocks) for brevity and to focus on Orthogonal Regularization and MDC blocks. We have added an ablation study on both a discriminative and generative task to evaluate the relative quality of these, as well as the Ternary adversarial loss. We also note that we are currently training a 40-layer, k=12 DenseNet (Huang et al.) with MDC blocks and Orthogonal Regularization that outperforms the D40K12 network reported in Huang et. al on CIFAR-100+ despite a <1% increase in parameters; we will update our table accordingly when training completes.
2)We recognize the frustration and difficulty in assigning quantitative metrics to image generation projects; in accordance with the suggestions of other reviewers, we have updated our ablation study to include the Inception score proposed by Salimans et. al in Improved-GAN, evaluated using a discriminative net trained on the CelebA binary attribute classification task. The Inception score has been found to correlate well with human perception (and we find that to be the case here as well, with a significant difference between our baseline VAE/GAN model and our most performant IAN model). We have also included two more measures of reconstruction accuracy (feature-wise loss and trait accuracy as evaluated by the classifier).
Conducting user studies is restrictive due to the nature of the funding for this project, which is orthogonal to the lead author's primary, sponsored work; We note that our primary goal is to improve the reconstruction capacity of the network, rather than the outright sample quality, which is what guided us towards a variety of measure of reconstruction metrics. Most of the recent work we are aware of that has performed such studies has done so using AMT, which we cannot afford, but we note that
3) We have updated the paper with a more thorough related work section that discusses the connection to iGAN in detail.
4) We have extended the appendix with a comparison to other latent-variable generative image models, and will update the appendix again once our CIFAR and tiny-ImageNet models finish training.
5) We recognize the split-brain nature of the paper, and have overhauled the paper to focus more on the specific contribution of the IAN, treating the Neural Photo Editor instead as an end-goal application, rather than as an analysis tool.
Though the NPE was originally developed for analyzing generative models, feedback since its release has indicated that fully-fledged generative image editing is a desirable user product, and we thus focus on tackling related design challenges.
mini-1) We  have updated the semi-supervised section and table to hopefully be more clear.
Thanks again for your in-depth comments!
Best,
Andy
UPDATE: The rewritten paper is more focused and precise than the previous version. The ablation studies and improved evaluation of the IAN model help to the make clear the relative contributions of the proposed MDC and orthogonal regularization.
Though the paper is much improved, in my opinion there is still too much emphasis on the photo-editing interface.
In addition, the MDC blocks are used in the generator of the model but their efficacy is measured via discriminative experiments. All-in-all I am updating my score to a 5.
This paper proposes a hybridization of a VAE and a GAN whereby the generator both generates random samples and produces reconstructions of the real data, and the discriminator attempts to classify each true data point, sample, and reconstruction as being real, fake, or reconstructed. It also proposes a user-facing interface with an interactive image editing algorithm along with various modifications to standard generative modeling architectures.
Pros:
+ The IAN model itself is interesting as standard GAN-based approaches do not simultaneously train an autoencoder.
Cons:
- The writing is unclear at times and the mathematical formulation of the IAN is not very precise.
- Many different ideas are proposed in the paper without sufficient empirical validation to characterize their individual contributions.
- The photo editing interface, though interesting, is probably better suited for a conference with more of an HCI focus.
* Section 2: The gradient descent step procedure seems to be quite similar to the approach proposed by [2]. More elaboration on the differences would be helpful.
* Section 3: It is unclear whether the discriminator has three binary outputs or if there is a softmax over the three possible labels. The paper does not mention whether L_G and L_D are minimized or maximized. Presumably they are both minimized, but in that case it is counter-intuitive that the generator attempts to decrease the probability that the discriminator assigns the "real" label to the generated samples and reconstructions.
In addition, in the minimization scenario L_D maximizes the probability of the correct labels being assigned to X_gen and \hat{X} but minimizes the probability of the "real" label being assigned to X.
* Section 3.2: It is odd that not training MADE leads to better performance, as training MADE should lead to a better variational approximation to the true posterior. More exploration seems warranted here.
* Section 3.3.2: One drawback of autoregressive approaches is that sampling is slow. How do you reconcile this with its use in an interactive application, where speed is important?
* Figure 7: The Inception score is typically expressed as an exponentiated KL-divergence. It is odd that the scores are being presented here as percents.
* Section 4.1: The Inception score's direct application to non-natural images is indeed problematic. One potential workaround is to compute exponentiated KL for a discriminative net trained specifically for the dataset, e.g. to predict binary attributes on CelebA.
Overall, the paper attempts to simultaneously do too many things. It could be made much stronger by focusing on the primary contribution, the IAN, and performing a comparison against similar approaches such as [1]. The other techniques, such as IAF with randomized MADE, MDC blocks, and orthogonal regularization are potentially interesting in their own right but the current results are not conclusive as to their specific benefits.
[1] Larsen, Anders Boesen Lindbo, Søren Kaae Sønderby, and Ole Winther. "Autoencoding beyond pixels using a learned similarity metric." arXiv preprint arXiv:1512.09300 (2015).
[2] J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros, “Generative Visual Manipulation on the Natural Image Manifold,” ECCV 2016.
Hi,
Thanks for the review! We have thoroughly revised the paper to take into account reviewer comments.
In particular, we have changed the focus of the paper to the specific design design challenges towards which the IAN is focused, and include the Neural Photo Editor as a proof-of-concept demonstrating the viability of our proposed method, in the hopes of mollifying the disjoint focus previously noted. We have also removed our section on IAF with randomized MADE and RGB/Beta blocks, and run a set of in-depth ablation studies on CIFAR-100 and CelebA to evaluate the relative usefulness of Orthogonal Regularization, our Multiscale Dilated Convolution blocks, and the Ternary Adversarial Loss. Due to time constraints, we were unable to re-run our 128x128 ImageNet experiments (which made use of IAF and RGB/Beta blocks), so we have withdrawn our ImageNet samples and reconstructions. We are currently training networks on tiny-Imagenet and CIFAR, and will hopefully update the paper when those have finished training.
-We have updated our formulation of the IAN to be more precise (in line with previous work, in particular (Larsen 2016)) and hopefully more clear.
-We have slimmed the paper (removing IAF with randomized MADE and RGB/Beta blocks, our two least important contributions) and added an in-depth ablation study to more thoroughly investigate the individual contributions of the proposed ideas.
-We have changed the focus of the paper towards the design challenges the IAN attempts to overcome, and treat the NPE as an end-goal application, rather than as a tool for analysis.
*Section 2: We have updated the paper with a more complete Related Work section, where we discuss the connection to iGAN in detail.
*Section 3: We have rewritten our formulation of the IAN to provide the relevant details to answer these questions.
*Section 3.2: We agree that the phenomenon is odd, but note that we consistently observed superior performance (both in reconstruction accuracy, stability,  and in our visual inspection of samples) when using an untrained MADE; we have yet to find a satisfying theoretical reason why this might be the case. We have since withdrawn this section and re-run our experiments to not include IAF, for the sake of brevity and to focus on our primary contributions.
*Section 3.3.2: Fully autoregressive approaches are restrictively slow, but in this case we are only sampling the R->G->B channels in sequence (rather than autoregressively sampling pixels a la PixelCNN). This is the equivalent of adding two slim extra layers on the end, which we found did not impact real-time speed even on a modest laptop GPU. We have since withdrawn the RGB/Beta blocks for brevity and to focus on our primary contributions.
*Figure 7: This was a typo which we have since fixed.
*Section 4.1: We have done just this, and trained a 40 Layer DenseNet (Huang, 2016) on the CelebA classification task, which we then use during our updated ablation study for evaluating the Inception score, as well as the feature-wise loss and the preservation of binary attributes.
Endnote: We have significantly revised the paper to juggle fewer plates, and we include a VAE/GAN baseline in all of our ablation studies.
Best,
Andy
Hi,
Thanks for the updated review! We would like to point out that the MDC blocks we use in the generator are indeed evaluated in our generative experiments as well as the discriminative experiments; their inclusion/exclusion is denoted in column 1 of Table 2.
Best,
Andy
The paper presents two main contributions:
(1) A novel model visualization and photo manipulation technique that allows to transform an image using a paintbrush, much like in an image editing software.
(2) A hybridization of GANs and VAEs called Introspective Adversarial Network.
The main problem I have with the paper is that it feels very much like two papers in one with a very loose story tying the two together.
On one hand, the neural photo editing technique is presented in sufficient detail to be reproducible and it is shown to be effective. I personally find the idea exciting, but in order for it to be of interest to the ICLR community I think more emphasis should be put on what insights such a technique allows to gain on trained models.
On the other hand, the IAF model is introduced, along with multiple network architecture modifications for improving its performance. One criticism that I have regarding the presentation is that it makes it hard to assign credit to individual ideas when they are presented in a "list of things to make it work" fashion. I would like to see more empirical results in that direction to help clear up things.
Overall I think the paper proposes interesting ideas, but given its lack of focus on a single, cohesive story I think it is not yet ready for publication.
UPDATE: The rating has been updated to a 6 following the authors' reply.
Hi,
Thanks for the review! We have thoroughly revised the paper to take into account reviewer comments.
In particular, we have changed the focus of the paper to the specific design design challenges towards which the IAN is focused, and include the Neural Photo Editor as a proof-of-concept demonstrating the viability of our proposed method, in the hopes of mollifying the disjoint focus previously noted. We have also removed our section on IAF with randomized MADE and RGB/Beta blocks, and run a set of in-depth ablation studies on CIFAR-100 and CelebA to evaluate the relative usefulness of Orthogonal Regularization, our Multiscale Dilated Convolution blocks, and the Ternary Adversarial Loss. We hope that these studies present a clearer picture as to the relative usefulness of the individual contributions.
Due to time constraints, we were unable to re-run our 128x128 ImageNet experiments (which made use of IAF and RGB/Beta blocks), so we have withdrawn our ImageNet samples and reconstructions. We are currently training networks on tiny-Imagenet and CIFAR, and will hopefully update the paper when those have finished training.
Best,
Andy
Thank you for the update.
The added emphasis on introspective adversarial networks in the experiments section is definitely a step is the right direction. Nitpick: in my opinion, the narrative of the neural photo editor being a proof-of-concept demonstrating the viability of IANs would be better served if the IAN section came before the neural photo editor section.
I feel like my criticism regarding the difficulty of assigning credit to the ideas introduced in the paper has been satisfyingly addressed by the updated version of the paper.
As such, I am updating my rating to a 6.
Hi,
We've added one final update, including our CIFAR-10 and Imagenet experiments in the appendix, along with a slight expansion of the related work section and some tweaks to the discussion of experiments.
Best,
Andy

Meta-review:
role: PC
data: comment: Here is a summary of strengths and weaknesses as per the reviews:
 
 Strengths
 Work/application is exciting (R3)
 Enough detail for reproducibility (R3)
 May provide a useful analysis tool for generative models (R1)
 
 Weaknesses
 Clarity of the IAN model - presentation is scattered and could benefit from empirical analysis to tease out which parts are most important (R3,R2,R1); AC comments that the paper was revised in this regard and R3 was satisfied, updating their score
 Lack of focus: is it the visualization/photo manipulation technique, or is it the generative model? (R3, R2)
 Writing could use improvement (R2)
 Mathematical formulation of IAN not precise (R2)
 
 The authors provided a considerable overhaul of the paper, re-organizing/re-focusing it in response to the reviews and adding additional experiments. 
 
 This, in turn, resulted in R1, R2 and R3 upgrading their scores. The paper is more clearly in accept territory, in the ACÕs opinion. The AC recommends acceptance as a poster.

Decision: Accept (Poster)