Controlling generative models with continuous factors of variations
Authors: Antoine Plumerault
Hervé Le Borgne
Céline Hudelot
Venue:  (2020)
Keywords: Generative models
factor of variation
GAN
beta-VAE
interpretable representation
interpretability

Abstract: Recent deep generative models can provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing. Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation. To overcome these major issues, very recent works have shown the interest of studying the semantics of the latent space of generative models. In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like position or scale of the object in the image. Our method is weakly supervised and particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations. We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.

Published as a conference paper at ICLR 2020
CONTROLLING GENERATIVE MODELS WITH CONTINU- OUS FACTORS OF VARIATIONS
Antoine Plumerault"i, Hervé Le Borgne', Céline Hudelot CEA, LIST, Laboratoire Analyse Sémantique Texte et Image, Gif-sur-Yvette, Yvette, F-91191 France i Université Paris-Saclay. CentraleSupélee, 91 190, Yiftsuu--vvtte, France.
ABSTRACT
Recent deep generative models are able to provide photo-realistic images as well as visual or textual content embeddings useful address various tasks of computer vision and natural language processing. Their usefulness is nevertheless often limited by the lack of control over the generative process the poor understanding of the learned representation. To overcome these major issues, very recent work has shown the interest studying the semantics of the latent space of generative models. In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which specific properties of the generated image like the position or scale of the object in the image. Our method does not require human annotations and is particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom color variations. We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.
to control precisely
position
position
Figure I: Images generated with our approach and BigGAN model Brook al. 2018 showing that the position of the object can be controlled within the image.
INTRODUCTION
With the success of recent generative models to produce high-resolution photo-realistic images (Karras) 2018 Brock et al. (2018) Razavi et al. 2019), increasing number of applications emerging, such as image in- in-painting, dataset- syythesss, and deep-fakes. However, the use of generative models is often limited by the lack of control over the generated images. More control could be used to

---

Published as a conference paper at ICLR 2020
improve existing approaches which aim at generating new training examples (Bowles al. 2018 by allowing the user to choose more specifie properties of the generated images.
First attempts in this direction showed that one modify an attribute of generated image by adding learned vector on its latent code (Radford et al. 2015 or by combining the latent code of two images (Karras et al.] 2018). Moreover, the study of the latent space of generative models provides insights about its structure which is of particular interest as generative models are also powerful tools to learn unsupervised data representations. For example. RRafford al. (2015) observed on auto-encoders trained on datasets with labels for some factors of variations, that their latent spaces exhibit a vector space structure where some directions encode the said factors of variations.
We suppose that images result from underlying factors of variation such as the presence of objects, their relative positions or the lighting of the scene. We distinguish two categories of factors of variations. Modal factors of variation are discrete values that correspond to isolated clusters in the data distribution, such as the category of the generated object. On the other hand, the size of an object or its position are described by Continuous factors of variations, expressed in range of possible values. As humans, we naturally describe images by using factors of variations suggesting that they are an efficient representation of natural images. For example, to describe a scene, one likely enumerates the objects seen, their relative positions and relations and their characteristics (Berg et al. 2012). This way of characterizing images is also described in Krishna et al. (2016). Thus, explaining the latent space of generative models through the lens of factors of variation is promising. However, the control over the image generation is often limited discrete factors and requires both labels and an encoder model. Moreover, for continuous factors of variations described by real parameter t, previous works do not provide a way to get precise control over f.
In this paper, we propose a method to find meaningful directions in the latent space of generative models that can be used to control precisely specific continuous factors of variations while the literature has mainly tackled semantic labeled attributes like gender, emotion or object category (Radford et al.) | Odena et al., 2016). We test our method on image generative models for three factors of variation of an object in an image: vertical position, horizontal position and scale. Our method has the advantage of not requiring a labeled dataset nor model with an encoder. It could be adapted to other factors of variations such as rotations, change of brightness, contrast, color or more sophisticated transformations like local deformations. However, we focused on the position and scale as these quantities that can be evaluated, allowing to measure quantitatively the effectiveness of our method. We demonstrate both qualitatively and quantitatively that such directions can be used to control precisely the generative process and show that method can reveal interesting insights about the structure of the latent space. Our main contributions are:
We propose method to find interpretable directions in the latent space of generative models, corresponding to parametrizable continuous factors of variations of the generated image.
We show that properties of generated images can be controlled precisely by sampling latent representations along linear directions.
We propose novel reconstruction loss for inverting generative models with gradient descent.
We give insights of why inverting generative models with optimization can be difficult by reasoning about the geometry of the natural image manifold.
We study the impacts of disentanglement the ability to control the generative models.
2 LATENT SPACE DIRECTIONS OF FACTOR OF VARIATION
We argue that it is easier to modify a property of an image than to obtain a label describing that property. For example, it is easier to translate an image than to determine the position of an object within said image. Hence, if we compute its difference with the latent code of the original image to find the direction in the latent space which corresponds to this specific transformation as in Radford et al. (2015).
determine the latent code of transformed image, we can
Let us consider a generative model G : x E 2 + I, with Z its latent space of dimension d and the space of images, and transformations Te I + I characterized by continuous parameter t. For example if T is a rotation, then could be the angle, and T is a translation, then t could be component of the vector of the translation in an arbitrary frame of reference. Let 2o be vector
2

---

Published as a conference paper at ICLR 2020
of 2 and I Gzz)) generated image. Given transformation Tr, aim at finding žy such that G( = (I) to then use the difference between 2y and 2r in order to estimate the direction encoding the factor of variation described by T.
2.1 LATENT SPACE TRAJECTORIES OF AN IMAGE TRANSFORMATION
Given an image / E I, we want to determine its latent code. When no encoder is available we can search an approximate latent code & that minimizes a reconstruction error L between / and I = G(2) (f can be seen as the projection of 7 on G(2)) i.e.
arg min C[I, G=)) #EZ
(1)
Solving this problem by optimization leads to solutions located in regions of low likelihood of the distribution used during training. It causes the reconstructed image / = G( to look unrealistic" Since follows a normal distribution N (0, Ia) in d-dimensional space, we have ||z|| ~ Xd- Thus, lim E [|] vd and lima-++oo Var (I|=ll) 0. Hence, when d large, the norm of is approximately equal to vd. This can be used to regularize the optimization by constraining x to verify | S dd:
= arg min C(I, G(z))
=E2. =Ilvvd
2.1.1 CHOICE OF THE RECONSTRUCTION ERROR C
One of the important choice regarding this optimization problem is that of C. In the literature, the most commonly used are the pixel-wise Mean Squared Error (MSE) and the pixel-wise cross-entropy as in Lipton & Tripathi (2017) and Creswell & Bharath (2016). However in practice, pixel-wise losses are known to produce blurry images. To address this issue, other works have proposed alternative reconstruction errors. However, they are based on an alternative neural network (Boesen Lindbo) Larsen et al., 2015) Johnson al., 2016) making them computationally expensive.
The explanation usually given for the poor performance of pixel-wise mean square error is that it favors the solution which is the expected value of all the possibilities (Mathieu et al. 2015 We propose to go deeper into this explanation by studying the effect of the MSE on images in the frequency domain. In particular, our hypothesis is that due to its limited capacity and the low dimension of its latent space, the generator dimensional. This uncertainty over texture configurations explains why textures are reconstructed as uniform regions when using pixel-wise emrors. In Appendix [ by expressing the MSE in the Fourier domain and assuming that the phase of high frequencies cannot be encoded in the latent space, we show that the contribution of high frequencies in such a loss is proportional pushing the optimization to solutions with less high frequencies, that is to say more blurry. In order to get sharper results we therefore propose to reduce the weight of high frequencies into the penalization of errors with the following loss:
not produce arbitrary texture patterns the manifold of textures is very high
their square magnitude
C(4,, = 12]4-11)F(o11 [1(4; 12) o
(3)
where F is the Fourier transform, * is the convolution operator and a is a Gaussian kernel. With a reduced importance given to the high frequencies to determine 2 when one this loss in equation 2 allows to benefit from a larger range of possibilities for G(z), including images with more details (i.e with more high frequencies) and appropriate texture to get more realistic generated images. A
qualitative comparison to some reconstruction errors and choices of a can be found in Appendix [ We also report quantitative comparison other losses, based on the Learned Perceptual Image Patch Similarity (LPIPS), proposed by Zhang et al. (2018).
We could have used L penalty the norm of to encode centered Gaussian prior the distribution of z. However the T,² penalty requires an additional hyper-paramete 'Indeed, if we model the value of pixel by a random variable x then arg min, E [x x} = [ In
that can he difficult to choose.
fact, this problem can easily generalized every pixel-wise loss if we assume that nearby pixels follows approximately the same distribution as arg min, TE. [Z,,, x)] will have the same value for nearby pixels.
3

---

Published as a conference paper at ICLR 2020
Algorithm 1: Create dataset of trajectories in the latent space which corresponds to transformation T in the pixel space. The transformation is parametrized by a parameter which controls degree of transformation. We typically use N 10 with (oSwS)) distributed regularly the interval [). Note that 2y and öt, are retained in D at each step to train the model of Section 2.2
Input: number of trajectories S, generator G, transformation function T, trajectories length N, threshold O
Result: dataset trajectories D
D - O for i E [1. do
20 NO I):
To < G(zn);
Z6t + 20
for n E [] do
Zát t arg min, [z)(2), Tat., (Io)):
if L(G(z) Tat, (l)) < θ then
D DU Eg. Sst, )),,)):
end
end
end
2.1.2 RECURSIVE ESTIMATION OF THE TRAJECTORY
our problem of finding zт such that G(ZT) 2 (I), given transformation Tr, can be solve through the following optimization problem:
Using equation 2
#T
min [(G((), =
(4)
In practice, this problem difficult and an "unlucky" initialization can lead to very slow convergence. Zhu et al. (2016) proposed to use an auxiliary network to estimate zr and use it as initialization. Training a specific network to initialize this problem nevertheless costly. One can easily observe that a linear combination of natural images is usually not a natural image itself, this fact highlights the highly curved nature the manifold of natural images in pixel space. In practice, the trajectories corresponding most transforms in pixel space may imply small gradients of the loss that slowdown the convergence of problem of Eq. 2 (see Appendix D). To address this, we guide the optimization on the manifold by decomposing the transformation Tr into smaller transformations [Tato...., Tst such that Tat,= Id and ó T and solve sequentially:
Zn arg min C (G (z; Zinit = Zn-1), .Ttt,, (G (zo))) for n 1,..., zEZ
(5)
each time initializing with the result of the previous optimization. In comparison to Zhu al. (2016), our approach does not require extra training and can thus be used directly without training new model. We compare qualitatively our method to naive optimization in Appendix
transformation on an image usually leads to undefined regions in the new image (for instance, for a translation to the right, the left hand side undefined). Hence, we ignore the value of the undefined regions of the image to compute L. Another difficulty is that often the generative model cannot produce arbitrary images. For example generative model trained on a given dataset not expected to be able to produce images where the object shape position is outside of the distribution of object shape positions in the dataset. This an issue when applying our method because as we generate images from random start point, we have no guarantee that the transformed images is still on the data manifold. To reduce the impact of such outliers, we discard latent codes that give a reconstruction error above threshold in the generated trajectories. In practice, remove one tenth of the latent codes which leads to the worst reconstruction errors. It finally results into Algorithm] to generate trajectories in the latent space.
4

---

Published as a conference paper at ICLR 2020
2.2 ENCODING MODEL OF THE FACTOR OF VARIATION IN THE LATENT SPACE.
After generating trajectories with Algorithm 1, we need to define a model which describes how factors of variations encoded in the latent space. We make the core hypothesis that the parameter I of a specific factor of variations can be predicted from the coordinate of the latent code along an axis u, thus we pose a model f Z + of the form t f(z) g((z, u)). with g R + R and ( the euclidean scalar product in R²
When g is monotonic differentiable function, we can without loss of generality, suppose that ||u]| 1 and that is an increasing function. Under these conditions, the distribution of = g((z, u)) when N (0, I) is given by 9 R R+:
(01) =
(L) dt
For example, consider the dSprite dataset (Matthey et al. 2017) and the factor corresponding to the horizontal position of object x in image, we have x that follows uniform distribution 4.5-0.5, 0.5]) in the dataset while the projection of onto an axis u follows a normal distribution (0, 1). Thus, is natural to adopt g
(-0.5, 0.5] and for g((z, u)):
d 9 (x)
(xxx (x,, | 0.5, 0.5.) (N (x):0, de
d
M122, u) ;0,
(g((z) ( u))
dx
1
d (12, u²) ((zz,));00)
d de g (g ui))
(z. (2.u) u)
g((z, u)) = erf
V2
However, in general, the distribution of the parameter is not known. One can adopt a more general parametrized model go of the form:
t f(o.w) go ((us 2)) with ||aa||
R and (6, u) trainable parameters of the model. We typically used piece-wise linear functions for ge.
with 9o :
However, this model cannot be trained directly as we not have access to / (in the case of horizontal translation the x-coordinate for example) but only to the difference 8t = (G(ss)) (G(=v) between an image G( and its transformation G(zzt) (5x or dy in the case of translation). We solve this issue by modeling öt instead of
öt = (io,)( - ((o) with ||/]|| = and go (0) =
Hence, u and are estimated by training f(), to minimize the MSE between &n and f(o. fie u.) (Zn) with gradient descent on a dataset produced by Algorithm for given transformation.
An interesting application of this method is the estimation of the distribution of the images generated by G by using E With the knowledge of go we can also choose how to sample images. For instance, let say that we want to have ~ @(t), with Φ IR R_ an arbitrary distribution, we can simply transform 2 ~ (0, 1) as follows:
f- (z, u) u + (Rep (12) u))u
(10)
with hss [0, 1] + IR and vb such that:
((x) -a)= J O(t:0,1)d hz)()) 1 (1) go (t)
(11)
These results are interesting to bring control not only on a single output of a generative model but also on the distribution of its outputs. Moreover, since generative models reflect the datasets which they have been trained, the knowledge of these distributions could be applied to the training dataset to reveal potential bias.
5

---

Published as a conference paper at ICLR 2020
3 EXPERIMENTS
Datasets: We performed experiments on two datasets. The first one is dSprites (Matthey et al. 2017 composed of 737280 binary x images containing a white shape on a dark background. Shapes can vary in position, scale and orientations making it ideal to study disentanglement. The second dataset is ILSVRC (Russakovsky al. 2013), containing 1.2M natural images from one thousand different categories.
Implementation details: All our experiments have been implemented with TensorFlow 2.0 (Abadi et al., 2015) and the corresponding code is available on github here We used a BigGAN model (Brock et al., 2018) whose weights are taken from TensorFlow-Hub allowing casy reproduction of our results. The BigGAN model takes two vectors inputs: latent vector
I 12 and
one-hot vector to condition the model to generate images from one category. The latent vector 2 is then split into six parts which are the inputs at different scale levels in the generator. The first part is injected at the bottom layer while next parts are used to modify the style of the generated image thanks to Conditional Batch Normalization layers (de Vries al. 2017). We also trained several 3- V-S (Higgins et al., 2017) to study the importance of seventanglement in the process of controlling generation. The exact B-VAE architecture used is given Appendix B The models were trained on dSprites (Matthey et al. 2017) with an Adam optimizer during 1e5 steps with a batch size of 128 images and learning rate 5.-4.
3.1 QUANTITATIVE EVALUATION METHOD
Evaluating quantitatively the effectiveness of our method on complex datasets is intrinsically difficult as it is not always trivial to measure a factor of variation directly. We focused our analysis on two factors of variations: position and scale. On simple datasets such as dSprites, the position of the object can be estimated effectively by computing the barycenter of white pixels. However, for natural images sampled with the BigGAN model, we have to use first saliency detection on the generated image to produce binary image from which
extract the barycenter. For saliency detection, we used the model provided by Hou et al. (2016) which is implemented in the PyTorch framework (Paszke al., 2017). The scale is evaluated by the proportion of salient pixels. The evaluation procedure is:
1. Get the direction u which should describe the chosen factor of variation with our method.
2. Sample latent codes from a standard normal distribution.
3. Generate images with latent code z (z, u.) u + tu with E TT.
4. Estimate the real value of the factor of variation for all the generated images.
5. Measure the standard deviation of this value with respect
Jahanian et (2019) proposed an alternative method for quantitative evaluation that relies on an object detector. Similarly to us, it allows an evaluation for and y shift as well as scale but is restricted to image categories that can be recognized by a detector trained on some categories of ILSVRC. The proposed approach is thus more generic.
3.2 RESULTS ON BIGGAN
We performed quantitative analysis on ten chosen categories of objects of ILSVRC, avoiding non actual objects such as "beach" or "cliff". Results are presented in Figure 2 (top). We observe that for the chosen categories of ILSVRC, we can control the position and scale of the object relatively precisely by moving along directions of the latent space found by our method.
However, one can still wonder whether the directions found are independent of the category of interest. To answer this question, we merged all the datasets of trajectories into one and learned common direction on the resulting datasets. Results for the ten test categories are shown in Figure 2 (bottom). This figure shows that the directions which correspond to some factors of variations are indeed shared between all the categories. Qualitative results are also presented in Figure for illustrative purposes. We also checked which parts of the latent code are used to encode position and scale. Indeed, BigGAN uses hierarchical latent code which means that the latent code is split into six parts

---

Published as a conference paper at ICLR 2020
badgo beathn
Kerpon gitrale
pideo
Figure 2: Quantitative results the categories of the ILSVRC dataset used for training (Top) and for ten other categories used for validation (Bottom) for three geometric transformations: horizontal and vertical translations and scaling. In blue, the distribution of the measured transformation parameter and in red the standard deviation of the distribution with respect to 4. Note that for large scales the algorithm seems to fail. However, this phenomenon very likely due the poor performance of the saliency model when the object of interest covers almost the entire image (scale 1.0). (best seen with zoom)
which are injected different level of the generator. We wanted to see by which part of the latent code these directions are encoded. The squared norm of each part of the latent code is reported in Figure 4 for horizontal position, vertical position and scale. This figure shows that the directions corresponding to spatial factors of variations are mainly encoded in the first part of the latent code. However, for the position, the contribution of level 5 is higher than for the x position and the scale. We suspect that it is due to correlations between the vertical position of the object in the image and its background that we introduced by transforming the objects because the background is not invariant by vertical translation because of the horizon.
3.3 THE IMPORTANCE DISENTANGLED REPRESENTATIONS
To test the effect of disentanglement on the performance of our method, we trained several 8- VAE (Higgins et al. 2017) on dSprites (Matthey et al. 2017), with different values. Indeed, 3-VAE are known for having more disentangled latent spaces as the regularization parameter 3 increases. Results can be seen in Figure 5 The figure shows that it is possible to control the position of the object on the image by moving the latent space along the direction found with our method. As expected, the effectiveness of the method depends on the degree of disentanglement of the latent space since the results are better with larger B3. Indeed the standard deviation decreases (red curve), allowing a more precise control the position of the generated images. This observation motivates further the interest of disentangled representations for control on the generative process.
Figure that 3 increases,
4 RELATED WORKS
Our work aims at finding interpretable directions the latent space of generative models to control their generative process. We distinguish two families of generative models: GAN-like models which
7

---

Published as a conference paper at ICLR 2020
T position
y position
ZOom
Figure 3: Qualitative results for some categories of ILSVRC dataset for three geometric transforma- tions: borizontal and vertical translations and scaling.
position
position
scale
ko41
Ind1
Ioal
lovis
Iwi11
Invia
Figure 4: Squared norm of each part of the latent code horizontal position, vertical position and scale.
do not provide an explicit way to get the latent representation an image and auto-encoders which provide encoder get the latent representation of images. From architectural point of view, conditional GANs (Odena et al. 2016 allows the user to choose the category of a generated object or some chosen properties of the generated image but this approach requires labeled dataset and use model which is explicitly designed to allow this control. Similarly regarding VAE, [Engel et al. (0018) identified that they suffer from trade-off between reconstruction accuracy and sample plausibility and proposed to identify regions of the latent space that correspond to plausible samples to improve reconstruction accuracy. They also use conditional reconstruction control the generative process. In comparison to these approaches, our method does not directly requires labels. With InfoGan, Chen al. (2016) shows that adding code the the input the GAN generator and optimizing with an appropriate regularization term leads to disentangle the latent space and make possible to find posteriori meaningfully directions. In contrast, we show that is possible to find such directions in several generative models, without changing the learning process (our approach could even be applied to InfoGAN) and with an a priori knowledge the factor of variation sought. More recently, Bau al. (2018) analyze the activations of the network's neurons to determine those that result in the presence of an object the generated image, and thus allows to control such a presence. In contrast, work focuses on the latent space and not on the intermediate activations inside the generator.
8

---

Published as a conference paper at ICLR 2020
3 = 1.0
3 = 5.0
8 = 10.0
B = 20.0
position
-0-
Figure 5: Results of our evaluation procedure with four 8-VAR for 3 = 1, 5, 10, 20. Note the erf shape of the results which indicates that the distribution of the shape positions has been correctly learned by the VAE, See Figure for additional information on how to read this figure.
One of our contribution and a part of our global method is a procedure to find the latent representation of an image when encoder is not available. Several previous works have studied how to invert the generator of a GAN to find the latent code of an image. Creswell & Bharath (2016) showed on simple datasets (MNIST (Lecun al., 1998) and Omniglot (Lake 2015)) that this inversion process can be achieved by optimizing the latent code to minimize the reconstruction error between the generated image and the target image. Lipton & Tripathi (2017) introduced tricks to improve the results on more challenging dataset (CelebA (Liu et al. 2015)). However we observed that these methods fail when applied on more complex datasets (ILSVRC (Russakovsky et al. 2015)). The reconstruction loss introduced in Section 2. is adapted to this particular problem and improves the quality of reconstructions significantly. We also theoretically justify the difficulties invert a generative model, compared to other optimization problems. In the context of vector space arithmetic in latent space, White (2016) argues that replacing a linear interpolation by a spherical one allows to reduce the blurriness well. This work also propose an algorithmic data augmentation, named "synthetic attribute", to generate image with less noticeable blur with a VAE. In contrast, we act directly the loss.
The closest works were released on ArXiv very recently (Goetschalckx et al. 2019 Jahanian et al. 2019) indicating that finding interpretable directions in the latent space of generative models to control their output is of high interest for the community. In these papers, the authors describe method to find interpretable directions in the latent space of the BigGAN model (Brock al. 2018). If their method exhibits similarities with ours (use of transformation, linear trajectories in the latent space), it also differs on several points. From a technical point of view our training procedure differs in the sense that we first generate a dataset of interesting trajectories to then train our model while they train their model directly. Our evaluation procedure is also more general as we use a saliency model instead of MobileNet-SSD v1 [iu et al. (2016) trained on specific categories of the ILSVRC dataset allowing us to measure performance on more categories. We provide additional insight on how auto-encoders can also be controlled with the method, the impact of disentangled representations on the control and on the structure of the latent space of BigGAN. Moreover we also propose an alternative reconstruction error to invert generators. However, the main difference we identify between the two works is the model of the latent space used. Our model allows a more precise control over the generative process and can be being adapted more cases.
9

---

Published as a conference paper at ICLR 2020
5 CONCLUSIONS
Generative models are increasingly more powerful but suffer from little control over the generative process and the lack of interpretability in their latent representations. In this context, we propose method to extract meaningful directions in the latent space of such models and use them to control precisely some properties of the generated images. We show that linear subspace of the latent space of BigGAN can be interpreted in term of intuitive factors of variation (namely translation and scale). It is an important step toward the understanding of the representations learned by generative models.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowiez, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL htttp::: /www. tensor www.orrrroww..ooo...o org/ Software available from tensorflow.org.
Olah,
David Bau, Jun- Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. GAN Dissection: Visualizing and Understanding Generative Adversarial Networks. arXiv e-prints, art. 11Xiv:1811.10597, 11 2018.
A. C. Berg, T. L. Berg, H. Daumé, J. Dodge, A. Goyal, X. Han, A. Mensch, M. Mitchell, A. Sood, K. Stratos, and K. Yamaguchi. Understanding and predicting importance in images. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 3562-3569, June 2012.
Anders Boesen Lindbo Larsen, Søren Kaac Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. arXiv e-prints, art. arXiv:1512.09300, Dec 2015.
Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Hammers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, and Daniel Rueckert. GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks. arXiv e-prints, art. 1rXiv:1810.1066 10866, Oct 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. CORR, abs/1809. 1009,, 2018. URL http://arriv..rr//aa0
1809.11099
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. arXiv e-prints, art. wtt.::1600036557. 06 2016.
Antonia Creswell and Anil Anthony Bharath. Inverting The Generator Of A Generative Adversarial Network. arXiv e-prints, art. arXiv:161 1.05644, 11 2016.
Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron Courville. Modulating early visual processing by language. arXiv e-prints, art arXiv: 1eXXiv1107.000 777700683. Jul 2017.
Jesse Engel, Matthew Hoffmann, and Adam Roberts. Latent constraints: Learning to generate conditionally from unconditional generative models. In International Conference on Learning Representations, 2018.
Lore Goetschalekx, Alex Andonian, Aude Oliva, and Phillip Isola. GANalyze: Toward Visual Definitions of Cognitive Image Properties. arXiv e-prints, art. arXiv:1906.1 10112, Jun 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representation, 2017.
10

---

Published as a conference paper at ICLR 2020
Qibin Hou, Ming-Ming Cheng, Xiao Wei Hu, Ali Borji, Zhuowen Tu, and Philip Torr. Deeply supervised salient object detection with short connections. arXiv e-prints, art. arXiv:1611.04499 11 2016.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. arXiv e-prints, art. arXiv: 1rX.v:1907.0 1907.07177 Jul 2019.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual Losses for Real-Time Style Transfer and Super-Resolutionn arXiv e-prints, art. arXiv: TttT:1603.08155 Mar 2016.
Tero Karras, Samuli Laine, and Timo Aila. Style-Based Generator Architecture for Generative Adversarial Networks. arXiv e-prints, art, arXiv:1812.04498,, 12 2018.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In arXiv preprint /rXiv:/602.073322 2016., 2016.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350662666 2015. ISSN 0036-8075. doi: 10.1 100116//ccieeccc.ab33500... /5C/6266/1332
URL
wcwp:ncceessiiicicmaea..ac htwp://cccmmm..ooc/oteeeeeeeee content/
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pp. 2278-2324, 1998.
Zachary C. Lipton and Subarna Tripathi. Precise Recovery of Latent Vectors from Generative Adversarial Networks. In ICLR workshop, 2017.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. European conference on computer vision, pp. 21-37. Springer, 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), 12 2015.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv e-prints, arXiv:151 1.1.014000000. Nov 2015.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http://ggihb..ccomeeepmmm////lld 2017.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional Image Synthesis With Auxiliary Classifier GANs. arXiv e-prints, art. arXiv:1610.09585, 10 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Dito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv e-prints, art. arXiv:1511.06434 11 2015.
Razavi, Aaron van den Oord, and Oriol Vinyals. Generating Diverse High-Fidelity Images with VQ-VAE-2. arXiv e-prints, art. arXiv: 9999000000666 Jun 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3 1>:11-2552, 2015. doi: 10.1007/s 1263-015-0816-2
Tom White. Sampling generative networks: Notes on a few effective techniques. CoRR, ha//1669.04665 2016. httpptt:://arx: wwtp://////aaaaaaoooooooo 609. 04468
11

---

Published as a conference paper at ICLR 2020
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. arXiv e-prints, art. arXiv:1801.03924 Jan 2018.
Zhou Wang, C. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from emror visibility to structural similarity. IEEE Transactions on Image Processing, 13(4))600-612, April 2004. doi: 10.1109/T1..2003.818999
Jun- Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative Visual Manipula- tion on the Natural Image Manifold. arXiv e-prints, art. arXiv:1609.03555 09 2016.
12

---

Published as a conference paper at ICLR 2020
A
PENALTY ON THE AMPLITUDE OF FREQUENCIES DUE TO MSE
In Section 2.1
we consider a target image / E I and generated image / = G(2) to be determined according to reconstruction loss L (Equation D. Let us note F(-) the Fourier transform. If L the usual MSF, from the Plancherel theorem, we have || r = || -[ Let us consider particular frequency w in the Fourier space and compute its contribution to the loss. The Fourier transform of I (resp. I) having a magnitude (resp. and a phase (resp. 6) at w, we have:
[F(1](W) [11 lie - ree012
(fcos(0) rccos())² + (fsin(0) rs)n
(12)
f2 ,²² - 2îr (oss(0)oos(()) + sin( sin sin(
R² ,² 2Fr cos(())oss(0) sin))) (inn)))
sin
If we model the disability of the generator to model every high frequency patterns as an uncertainty on the phase of high frequency of generated image, i.e by posing 0 ~ (((1, 2m)), the expected value of the high frequency contributions to the loss is equal to:
E
= + - 2Fr
E cos( cos(O) + (i) sin(O)
(13)
62 n2
The term r² is a constant w.r.t the optimization of L and can thus be ignored. The contribution to total loss C thus directly depends on j². While minimizing L, the optimization process tends to favour images f G(2) with smaller magnitudes in the high frequencies, that to say smoother images, with less high frequencies.
B B- VAE ARCHITECTURE
The 3- 3-VAE framework was introduced by Higgins et al. (017) to discover interpretable factorised latent representations for images without supervision. For our experiments, we designed a simple convolutional VAE architecture to generate images of size 64x64, the decoder network is the opposite of the encoder with transposed convolutions.
Encoder
Decoder
Convolution + ReLU
Dense + ReLU
filters=32 size=4 stride=2 pad=SAME
units=256
Convolution + ReLU
Dense + ReLU
filters=32 size=4 stride=2 pad=SAME
units=256
Convolution + ReLU
Reshape
filters=32 size=4 2 pad=SAME
shape=4x4x32
Convolution + ReLU
Transposed Convolution + ReLU
filters=32 size=4 stride=2 pad=SAME
filters=32 stid== pad
Dense ReLUU
Transposed Convolttion ReLU
units=256
filter=32 strne=33i-2zz44rride==22222 stride-tii pad
Dense ReLU
Transposed Convolution C ReLU
units=256
filter=32 size=4 stride=2
Pad= paadast-aaa--a
μr: Dense + Identity
Transposed Convolution Sigmoid
a: Dense + Exponential
filters=1 size=4 stride=2 pad=SAME
units=10
Table 1: VAE architecture used during experiments with the dSprites dataset.
13

---

Published as a conference paper at ICLR 2020
C QUALITATIVE AND QUANTITATIVE EXPERIMENTS WITH OUR RECONSTRUCTION ERROR
o=3
Figure 6: Reconstruction results with different values. We typically used standard deviation of pixels for the kernel.
With unconstrained
Target image
With ||||| S Vd
MSE
DSSIM
Our (a 5)
Figure 7: Reconstruction results obtained with different reconstruction errors: MSE, DSSIM (Zhou
Wang et al. 2004) and our loss. With or without the constraint on ||=||. Note the artifacts when using loss without constraining (best seen with zoom).
On Fig. 6 we show qualitative reconstruction results with our method (Eq. 3 for several values of a. On this representative example, observe quite good results with = 3 and penalizes too low frequencies that lead to a less accurate reconstruction.
Higher values
We also illustrate on Fig. a comparison of our approach to two others, namely classical Mean Square Error (MSE) and Structural dissimilarity (DSSIM) proposed by Zhou Wang et al. (2004). Results are also presented with an unconstrained latent code during optimization (Eq. [1] and the approach proposed (Eq. 2 This example show the accuracy of the reconstruction obtained with our approach, as well as the fact that the restriction of to ball of radius vd avoids the presence of artifacts.
We also performed quantitative evaluation of the performance of our approach. We randomly selected one image for each of the 1000 categories of the ILSVRC dataset and reconstructed it with our method with a budget of 3000 iterations. We then computed the Learned Perceptual Image Patch Similarity (LPIPS), proposed by Zhang et al. (2018), between the final reconstruction and the target image. We used the official implementation reported in Table 2 It suggests that images reconstructed using perceptually closer to the target image than those obtained with MSE or DSSIM. The higher standard deviation for the MSE reconstructed image I.IIS suggests that some images downgraded in
the LPIPS paper with default parameters. Results reconstruction enror
14

---

Published as a conference paper at ICLR 2020
terms of perception. It can be the case for the textured ones in particular, for the reasons explained in the [ection[A]
reconstruction error
mean LPIPS
std LPIPS
MSE
0.57
0.14
DSSIM
0.58
0.12
Our (ar
0.52
0.12
Table 2: Perceptual similarity measurements between image and its reconstruction for different reconstruction errors.
D ON THE DIFFICULTY OF OPTIMISATION ON THE NATURAL IMAGE MANIFOLD.
The curvature of the natural image manifold makes the optimisation problem of Equation 2 difficult to solve. This is especially true for factors variation which correspond to curved walks in pixel-space (for example translation rotation by opposition brightness contrast changes which are linear).
To illustrate this fact, we show that the trajectory described by an image undergoing common transformations is curved in pixel space. We consider three types of transformations, namely translation, rotation and scaling, and get images from the dSprites (Matthey et al., (017) dataset which correspond to the progressive transformation (interpolation) of image. To visualize, compute the PCA of the resulting trajectories and plot the trajectories on the two main axes of the PCA. The result this experiment can be seen in Figure g
Translation
Scale
Rotation
gan Bad
PCA
PCA
PCA
PCA
Figure 8: Two trajectories shown in the pixel space, between image and its transformed version, for three types transformations: translation, scale and orientation. Red: shortest path (interpolation) between the two extremes of the trajectory. Blue: trajectory of the actual transformation. At each position along the trajectories, we report the corresponding image (best seen with zoom).
In this figure, we can see that for large translations, the direction of the shortest path between two images in pixel-space is near orthogonal to the manifold. The same problem occurs for rotation and, smaller extent, for scale. However this problem does exist for brightness for example, as its change is linear transformation in pixel-space. This is problematic during optimization of the latent code because the gradient of the reconstruction loss with respect the generated image is tangent to this direction. Thus, when are in the case of near orthogonality, the gradient of the error with respect to the latent code small.
Indeed, let consider an ideal case where G is a bijection between 2 and the manifold of natural images. Let be z E Z, a basis of vectors tangent to the manifold at point G(z) is given by 3G(z)
(G(=)
If V C(G(())((()) Aunges) is near orthogonal to the manifold then:
aG(z),
iA 1, d: (G(()) T Iunga);
e; with c; (14)
0z,
15

---

Published as a conference paper at ICLR 2020
Thus,
aG(z)* Oz
V (urge))
(G
(15)
shows that when the direction of descent in pixel space is near orthogonal to manifold described by the generative model, optimization gets slowed down and can stop if the gradient of the loss with respect to the generated image orthogonal to the manifold.
For example, let assume have ideal GAN which generates small white circle black background, with a latent space of dimension that encodes the position of the circle. Let consider a generated image with the circle on the left of the image and we want move it to the right. Obviously, we thus have V=] - (T-((((z1)) ) O if the intersection of the two circles is empty (see Figure 8 since small translation of the object does not change the reconstruction error.
E ADDITIONAL QUALITATIVE EXAMPLES
position
y position
Z00m
brightness
Figure 9: Qualitative results for 10 categories of ILSVRC dataset for three geometric transformations
(horizontal and vertical translations and scaling) and for brightness.
16

---

Published as a conference paper at ICLR 2020
We show qualitative examples for images generated with the BigGAN model for position, scale and brightness. The images latent codes are sampled in the following way: 2 (z. u) u + au with [-3.3] 3] and the learned direction. We have chosen the categories to produce interesting results: for position and scale categories are objects, for brightness categories are likely to be seen a bright dark environment. Notice that for some of the chosen categories, failed to control the brightness of the image. It is likely due to the absence of dark images for these categories in the training data. for position and scale, the direction learned on the ten categories presented here while for brightness only the five top categories are used.
F QUALITATIVE COMPARISON BETWEEN OUR OPTIMIZATION METHOD AND THE NAIVE METHOD.
0
10
20
30
40
50
60
70
80
90
100
500
850
1000
1500
2000
2500
3000
3500
4000
4500
Figure 10: Comparison of the speed of convergence on single example for our method (top) given by equation Sand and a naive approach (bottom) given by equation The numbers indicate the step of optimization. Both experiences have been conducted with Adam optimizer with a learning rate of le-1.
17


Reviews:
[{"review_id": "r1lnq4gCFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an algorithm to find linear trajectories in the latent space of a generative model that correspond to a user-specified transformation T in image space. Roughly, the latent trajectory is obtained by inverting the generator at the transformed image and a clever recursive estimation strategy is proposed to overcome difficulties in this nonconvex optimization. Qualitative results of the method, applied to a (pretrained) BigGAN model are shown, where the transformations are chosen as translation, zoom or brightness. A quantitative evaluation is performed on the dSprites and ILSVRC dataset. \n\nMy take:\nThere seem to be a lot of errors and typos in the manuscript, which made the paper unfortunately a bit frustrating to review. In particular, I had trouble following and understanding the details of the main procedure used to obtain the linear latent trajectories. Considering the recent works (Goetschalckx et al, 2019) and (Jahanian et al, 2019), I also don't see too much novelty in this approach. Therefore, I cannot recommend acceptance of this paper at this point. \n\nDetails:\n\n1) In algorithm 1, there seem to be some typos which makes it difficult understand the method in detail. z_{\\delta t} is initialized as z_0 and then never changed but always appended into the data set. Should it maybe be z_{\\delta t} <- argmin ... instead of z_t <- argmin ... ? But then why initialize z_{\\delta t} at all? Why store tuples of three values in D, especially store z_0 multiple times?\n\nWhile it is clear, formally the method always discards D and one might add a D_i <- D at the end.\n\n2) I could not follow the reasoning in Section 2.2 and the clarity should be improved, as it is one of the main contributions of the work. In particular, I would like to see the intuition behind the model t = g(<u, z>) better described. \n\nWhy does the projection of z follow a normal distribution? Is it because the latent distribution in the GAN is chosen as a normal distribution?  \n\nWhat is the loss for training f_{\\theta,u}? How is the dataset D used here? \n\nMinor comments / typos / suggestions (no influence on my rating):\n- InfoGAN (Chen et al., 2016) does not require a labeled dataset, the corresponding sentence in related work should be reformulated a bit. \n- Please use \\operatorname or \\text in math mode for operators such as Var or text.  \n- 'Encodes a the parameter t' --> 'Encodes the parameter t'; Many other typos, please run a spell checker.\n- For image translation, what boundary conditions are used? A sensible way would be to impose the reconstruction loss not on the full image but only on the smaller part. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgA8ZLKYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to learn and control continuous factors of variations within generative models by finding meaningful directions in the latent space which correspond to specified properties. A new method is proposed for inverting generative models and embedding images in the latent space when an encoder is not available. Specifically, reconstruction error is defined in the Fourier domain such that the weighting on high frequency image components can be reduced. Results are evaluated with qualitative comparison to previous embedding methods. Using this image embedding technique, a dataset of latent space trajectories is created by manipulating a desired property in images (such as position or scale) via affine transformations and recording the latent space vectors of the original and new images. The dataset is then used to learn a simple model of the latent space transformation corresponding to changes in the desired image property, which in turn can be used to manipulate images accordingly. To evaluate the effectiveness of this image manipulation approach, a saliency detector is used to measure the change in position or scale of objects in generated images as the latent codes are changed.\n\nOverall, I would tend towards accepting this work. The goal of being able to manipulate continuous factors of variation within generative models is useful for controllable image synthesis, and the proposed method clearly achieves the desired result.\n\n\nThings to improve the paper:\n1) The paper proposes a new reconstruction error metric which is optimized to embed images into the latent space of the generative models. While this new metric is compared qualitatively to existing methods, quantitative evaluation is lacking. It would be useful to also include quantitative comparison of methods measuring the perceptual distance between the original image and the embedded image, perhaps by using Learned Perceptual Image Patch Similarity (LPIPS) [1].\n\n\nMinor things to improve the paper that did not impact the score:\n2) In the abstract: \"Our method is weakly supervised...\". I am not sure if this method would be considered weakly supervised. I might tend more towards calling it self-supervised, since we have exact labels that are derived from transformations applied to the images themselves.\n\n3) In the first paragraph of the introduction: \"an increasing number of applications are emerging such as image in-painting, dataset-synthesis, deep-fakes... \". I find the use of the ellipses here to be a bit strange, since it seems like the sentence is trailing off mid-thought. I would recommend the use of \"etc.\" over \"...\".\n\n4) In Section 2.2, second paragraph, the dSprite dataset is mentioned but not cited. The reference is not given until Section 3. Should the citation be paired with the first mention of the dataset? Or even just in both places.\n\n5) In Section 3, Implementation details: \"The first part is injected at the bottom layer while next parts are used to modify the style of the generated image thanks to AdaIN layers (Huang & Belongie, 2017)\". BigGAN uses conditional BatchNorm instead of AdaIN, although they are both very similar. I think the proper citation here is [2], which first introduced conditional BatchNorm.\n\n\nQuestions:\n6) I am not fully convinced of the argument that using a saliency detector makes the method more general purpose than a dedicated object detector. The majority of high quality generative models are class conditional, hence requiring a labelled dataset, and therefore an object detector can easily be trained on the same dataset. Additionally, Section 3.2 mentions that \"We performed quantitative analysis on ten chosen categories for which the object can be easily segmented by using saliency detection approach\", which seems to indicate that the saliency detector struggles with some objects. How does the saliency detector perform on more complicated objects?\n\n\nReferences:\n[1] Zhang, Richard, et al. \"The unreasonable effectiveness of deep features as a perceptual metric.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n\n[2] De Vries, Harm, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C. Courville. \"Modulating early visual processing by language.\" In Advances in Neural Information Processing Systems, pp. 6594-6604. 2017.\n\n\n### Post-Rebuttal Comments ###\nThanks you for addressing my concerns and for adding the quantitative reconstructions measures. Appendix C looks much more complete now. My overall opinion of the paper remains about the same, so I will leave my score unchanged.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HkxRzJWpFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper proposes methods to find interpretable vectors in the latent space of generative models (similar to finding Smile Vectors [White, 2016]) which control simple object transformations like zoom or translation. The basic idea is that so long as one can apply the desired transformation to an image, one can solve for the latent which minimizes the reconstruction between G(z) and the transformed image; doing this for various parameters of the transformation (i.e. different levels of zoom, varying amounts of brightening or translation) allows one to learn a parametric mapping specifying how to vary the latent to achieve the desired output change. The authors make several changes to the naïve optimization procedure of vanilla SGD, most notably using reconstruction error on Gaussian-blurred images to encourage matching of low-frequency features rather than high-frequency features. The resulting framework is applied to an ImageNet GAN for a variety of transformation, producing results which qualitatively and quantitatively indicate that the method works for the shown transformations, along with some analysis of the behavior of the model.\n\nMy take:\n\nThis is a well-reasoned and well-presented paper following in the spirit of Smile Vector type investigations, with compelling results. The core idea is simple, and I like that it doesn’t require human labeling: one merely needs to be able to simulate some approximation of the desired transform, and one can find the latent space trajectory that corresponds to the model’s approximation of that transform. I think this is promising  next step in this area (there have been a few papers very recently on it, so I think improving constraints and control of generative models is getting a decent amount of intention) and is worthy of acceptance at ICLR2020 (7/10; reasonably clear accept).\n\nNotes:\n\n-“Sampling Generative Models” (White, 2016, https://arxiv.org/abs/1609.04468) should be cited and discussed, and ideally so should “Latent constraints: Learning to generate conditionally from unconditional generative models” (Engel et al, 2017, https://arxiv.org/abs/1711.05772)--both are quite relevant IMO.\n\nMinor:\n\nThe first sentence ends with an ellipsis. Is this intentional or a draft holdover? Either way I think it should at least be replaced with an ‘etc’ or ideally an oxford comma and an ‘and’. \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following the revision and the discussion, all three reviewers agree that the paper provides an interesting contribution to the area of generative image modeling. Accept.
Decision: Accept (Poster)