Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes an algorithm to find linear trajectories in the latent space of a generative model that correspond to a user-specified transformation T in image space. Roughly, the latent trajectory is obtained by inverting the generator at the transformed image and a clever recursive estimation strategy is proposed to overcome difficulties in this nonconvex optimization. Qualitative results of the method, applied to a (pretrained) BigGAN model are shown, where the transformations are chosen as translation, zoom or brightness. A quantitative evaluation is performed on the dSprites and ILSVRC dataset. 

My take:
There seem to be a lot of errors and typos in the manuscript, which made the paper unfortunately a bit frustrating to review. In particular, I had trouble following and understanding the details of the main procedure used to obtain the linear latent trajectories. Considering the recent works (Goetschalckx et al, 2019) and (Jahanian et al, 2019), I also don't see too much novelty in this approach. Therefore, I cannot recommend acceptance of this paper at this point. 

Details:

1) In algorithm 1, there seem to be some typos which makes it difficult understand the method in detail. z_{\delta t} is initialized as z_0 and then never changed but always appended into the data set. Should it maybe be z_{\delta t} <- argmin ... instead of z_t <- argmin ... ? But then why initialize z_{\delta t} at all? Why store tuples of three values in D, especially store z_0 multiple times?

While it is clear, formally the method always discards D and one might add a D_i <- D at the end.

2) I could not follow the reasoning in Section 2.2 and the clarity should be improved, as it is one of the main contributions of the work. In particular, I would like to see the intuition behind the model t = g(<u, z>) better described. 

Why does the projection of z follow a normal distribution? Is it because the latent distribution in the GAN is chosen as a normal distribution?  

What is the loss for training f_{\theta,u}? How is the dataset D used here? 

Minor comments / typos / suggestions (no influence on my rating):
- InfoGAN (Chen et al., 2016) does not require a labeled dataset, the corresponding sentence in related work should be reformulated a bit. 
- Please use \operatorname or \text in math mode for operators such as Var or text.  
- 'Encodes a the parameter t' --> 'Encodes the parameter t'; Many other typos, please run a spell checker.
- For image translation, what boundary conditions are used? A sensible way would be to impose the reconstruction loss not on the full image but only on the smaller part. 

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Summary: 

This paper proposes methods to find interpretable vectors in the latent space of generative models (similar to finding Smile Vectors [White, 2016]) which control simple object transformations like zoom or translation. The basic idea is that so long as one can apply the desired transformation to an image, one can solve for the latent which minimizes the reconstruction between G(z) and the transformed image; doing this for various parameters of the transformation (i.e. different levels of zoom, varying amounts of brightening or translation) allows one to learn a parametric mapping specifying how to vary the latent to achieve the desired output change. The authors make several changes to the naïve optimization procedure of vanilla SGD, most notably using reconstruction error on Gaussian-blurred images to encourage matching of low-frequency features rather than high-frequency features. The resulting framework is applied to an ImageNet GAN for a variety of transformation, producing results which qualitatively and quantitatively indicate that the method works for the shown transformations, along with some analysis of the behavior of the model.

My take:

This is a well-reasoned and well-presented paper following in the spirit of Smile Vector type investigations, with compelling results. The core idea is simple, and I like that it doesn’t require human labeling: one merely needs to be able to simulate some approximation of the desired transform, and one can find the latent space trajectory that corresponds to the model’s approximation of that transform. I think this is promising  next step in this area (there have been a few papers very recently on it, so I think improving constraints and control of generative models is getting a decent amount of intention) and is worthy of acceptance at ICLR2020 (7/10; reasonably clear accept).

Notes:

-“Sampling Generative Models” (White, 2016, https://arxiv.org/abs/1609.04468) should be cited and discussed, and ideally so should “Latent constraints: Learning to generate conditionally from unconditional generative models” (Engel et al, 2017, https://arxiv.org/abs/1711.05772)--both are quite relevant IMO.

Minor:

The first sentence ends with an ellipsis. Is this intentional or a draft holdover? Either way I think it should at least be replaced with an ‘etc’ or ideally an oxford comma and an ‘and’. 


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method to learn and control continuous factors of variations within generative models by finding meaningful directions in the latent space which correspond to specified properties. A new method is proposed for inverting generative models and embedding images in the latent space when an encoder is not available. Specifically, reconstruction error is defined in the Fourier domain such that the weighting on high frequency image components can be reduced. Results are evaluated with qualitative comparison to previous embedding methods. Using this image embedding technique, a dataset of latent space trajectories is created by manipulating a desired property in images (such as position or scale) via affine transformations and recording the latent space vectors of the original and new images. The dataset is then used to learn a simple model of the latent space transformation corresponding to changes in the desired image property, which in turn can be used to manipulate images accordingly. To evaluate the effectiveness of this image manipulation approach, a saliency detector is used to measure the change in position or scale of objects in generated images as the latent codes are changed.

Overall, I would tend towards accepting this work. The goal of being able to manipulate continuous factors of variation within generative models is useful for controllable image synthesis, and the proposed method clearly achieves the desired result.


Things to improve the paper:
1) The paper proposes a new reconstruction error metric which is optimized to embed images into the latent space of the generative models. While this new metric is compared qualitatively to existing methods, quantitative evaluation is lacking. It would be useful to also include quantitative comparison of methods measuring the perceptual distance between the original image and the embedded image, perhaps by using Learned Perceptual Image Patch Similarity (LPIPS) [1].


Minor things to improve the paper that did not impact the score:
2) In the abstract: "Our method is weakly supervised...". I am not sure if this method would be considered weakly supervised. I might tend more towards calling it self-supervised, since we have exact labels that are derived from transformations applied to the images themselves.

3) In the first paragraph of the introduction: "an increasing number of applications are emerging such as image in-painting, dataset-synthesis, deep-fakes... ". I find the use of the ellipses here to be a bit strange, since it seems like the sentence is trailing off mid-thought. I would recommend the use of "etc." over "...".

4) In Section 2.2, second paragraph, the dSprite dataset is mentioned but not cited. The reference is not given until Section 3. Should the citation be paired with the first mention of the dataset? Or even just in both places.

5) In Section 3, Implementation details: "The first part is injected at the bottom layer while next parts are used to modify the style of the generated image thanks to AdaIN layers (Huang & Belongie, 2017)". BigGAN uses conditional BatchNorm instead of AdaIN, although they are both very similar. I think the proper citation here is [2], which first introduced conditional BatchNorm.


Questions:
6) I am not fully convinced of the argument that using a saliency detector makes the method more general purpose than a dedicated object detector. The majority of high quality generative models are class conditional, hence requiring a labelled dataset, and therefore an object detector can easily be trained on the same dataset. Additionally, Section 3.2 mentions that "We performed quantitative analysis on ten chosen categories for which the object can be easily segmented by using saliency detection approach", which seems to indicate that the saliency detector struggles with some objects. How does the saliency detector perform on more complicated objects?


References:
[1] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

[2] De Vries, Harm, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C. Courville. "Modulating early visual processing by language." In Advances in Neural Information Processing Systems, pp. 6594-6604. 2017.


### Post-Rebuttal Comments ###
Thanks you for addressing my concerns and for adding the quantitative reconstructions measures. Appendix C looks much more complete now. My overall opinion of the paper remains about the same, so I will leave my score unchanged.

review_title: Official Blind Review #3

Meta-review: Following the revision and the discussion, all three reviewers agree that the paper provides an interesting contribution to the area of generative image modeling. Accept.
Decision: Accept (Poster)