On the "steerability" of generative adversarial networks
Authors: Ali Jahanian*
Lucy Chai*
Phillip Isola
Venue:  (2020)
Keywords: generative adversarial network
latent space interpolation
dataset bias
model generalization

Abstract: An open secret in contemporary machine learning is that many models work beautifully on standard benchmarks but fail to generalize outside the lab. This has been attributed to biased training data, which provide poor coverage over real world events. Generative models are no exception, but recent advances in generative adversarial networks (GANs) suggest otherwise -- these models can now synthesize strikingly realistic and diverse images. Is generative modeling of photos a solved problem? We show that although current GANs can fit standard datasets very well, they still fall short of being comprehensive models of the visual manifold. In particular, we study their ability to fit simple transformations such as camera movements and color changes. We find that the models reflect the biases of the datasets on which they are trained (e.g., centered objects), but that they also exhibit some capacity for generalization: by "steering" in latent space, we can shift the distribution while still creating realistic images. We hypothesize that the degree of distributional shift is related to the breadth of the training data distribution. Thus, we conduct experiments to quantify the limits of GAN transformations and introduce techniques to mitigate the problem.   Code is released on our project page: https://ali-design.github.io/gan_steerability/

Published as a conference paper at ICLR 2020
ON THE "STEERABILITY" OF GENERATIVE ADVERSARIAL NETWORKS
Ali Jahanian Lucy Chai & Phillip Isola Massachusetts Institute of Technology Cambridge, MA 02139, USA (jahanian, Irchai, phi 11 11 p p@mit Comit.eds edu
ABSTRACT
An open secret in contemporary machine learning that many models work beau- tifully on standard benchmarks but fail to generalize outside the lab. This has been attributed to biased training data, which provide poor coverage over real world events. Generative models are no exception, but recent advances in generative adversarial networks (GANs) suggest otherwise - these models can now synthe- size strikingly realistic and diverse images. Is generative modeling of photos a solved problem? We show that although current GANs can fit standard datasets very well, they still fall short of being comprehensive models of the visual man- ifold. In particular, we study their ability to fit simple transformations such as camera movements and color changes. We find that the models reflect the biases of the datasets on which they are trained (e.g., centered objects), but that they also exhibit some capacity for generalization: by "steering" in latent space, we can shift the distribution while still creating realistic images. We hypothesize that the degree of distributional shift related to the breadth of the training data distribu- tion. Thus, we conduct experiments to quantify the limits of GAN transformations and introduce techniques to mitigate the problem. Code released on our project lilignnnttaaaaraa
page: https::///// //al des i gn. www.ggituub........
INTRODUCTION
The quality of deep generative models has increased dramatically the past few years. When introduced in 2014, Generative Adversarial Networks (GANs) could only synthesize MNIST digits and low-resolution grayscale faces (Goodfellow al. 2014).. The most recent models, however, produce diverse high-resolution images that are often indistinguishable from natural photos (Brock)
al. 2018 Karras et al., 2018).
Science fiction has long dreamed of virtual realities filled of synthetic content as rich as, or richer, than the real world (e.g.. The Matrix, Ready Player One). How close are we to this dream? Tra- ditional computer graphics can render photorealistic 3D scenes, but cannot automatically generate detailed content. Generative models like GANs, in contrast, can create content from scratch, but we do not currently have tools for navigating the generated scenes in the same kind of way as you can walk through and interact with 3D game engine.
In this paper, we explore the degree to which you can navigate the visual world of GAN. Figure illustrates the kinds of transformations we explore. Consider the dog at the top-left. By moving in some direction of GAN latent space, can we hallucinate walking toward this dog? As the figure indicates, and as we will show in this paper, the answer is yes. However, as we continue to zoom in, quickly reach limits. Once the dog face fills the full frame, continuing to walk in this direction fails to increase the zoom. A similar effect occurs in the daisy example (row of Fig. [T. where a direction in latent space moves the daisy up and down, but cannot move out of frame.
We hypothesize that these limits are due to biases in the distribution of images on which the GAN is trained. For example, if the training dataset consists of centered dogs and daises, the same may be the case in GAN -eNegreee images. Nonetheless, we find that some degree of transformation is achieve certain transformations but not others?
possible. When and why

---

Published as a conference paper at ICLR 2020
Zoom
Brightness
Shift Y
Rotate2I>
Shift X
Rotate3
Figure 1: Learned latent space trajectories in generative adversarial networks correspond visual transformations like camera shift and zoom. Take the "steering wheel", drive the latent space, and explore the natural image manifold via generative transformations!
This paper seeks to quantify the degree to which achieve basic visual transformations by navigating in GAN latent space. In other words, are GANs "steerable" in latent space' [ We analyze the relationship between the data distribution achieving these transformations. From our experiments, it is possible to shift the distribution of generated images
which the model is trained and the success in
degree, but cannot extrapolate entirely out of the dataset's support. In particular, attributes can be shifted in proportion to the variability of that attribute in the training data. We further demonstrate an approach to increase model steerability by jointly optimizing the generator and latent direction, together with data augmentation current criticisms of generative models is that they simply interpolate between datapoints, and fail to generate anything truly new, but our results add nuance to this story. It is possible to achieve distributional shift, but the ability to create realistic images from a modified distributions relies on sufficient diversity in the dataset along the dimension that we vary.
training images. One of the
Our main findings are:
A simple walk in the latent space of GANs achieves camera motion and color transforma- tions in the output image space. These walks are learned in self-supervised manner without labeled attributes or distinct source and target images.
The linear walk is as effective more complex non-linear walks, suggesting that the mod- els learn to roughly linearize these operations without being explicitly trained to do so.
The extent each transformation is limited, and we quantify a relationship between dataset variability and how much
shift the model distribution.
The transformations tectures, e.g. BigGAN, StyleGAN, and DCGAN, and illustrate different disentanglement properties in their respective latent spaces.
general-puppe purposs framework that work with different model archi-
Data augmentation improves steerability, as does jointly training the walk trajectory and the generator weights, which allows us to achieve larger transformation effects.
RELATED WORK
Latent space manipulations can be seen from several perspectives - how we achieve it, what limits it, and what it enables to do. Our work addresses these three aspects together, and briefly refer to each one in related work.
Interpolations in latent space Traditional approaches to image editing with GAN latent spaces find linear directions that correspond to changes in labeled attributes. such as smile-vectors and gender-vectors for faces (Radford al. 2015 Karras et al. 2018). However these manipulations are not exclusive to GANs; in flow-based generative models, linearly interpolating between two encoded images allow one to edit source image toward attributes of the target (Kingma & Dhari- wal 2018). Möllenhoff & Cremers (2019) proposes modified GAN formulation by treating data
We the "steerable" in analogy the classic steerable filters of Freeman Adelson 1991
2

---

Published as a conference paper at ICLR 2020
as directional k-currents, where moving along tangent planes naturally corresponds to interpretable manipulations, [Upchurch et al. (2017) removes the generative model entirely and instead interpo- lates in the intermediate feature space of pretrained classifier, again using feature mappings of source and target sets to determine an edit direction. Unlike these approaches, we learn our latent- space trajectories in self-suuprvised manner without labeled attributes or distinct source and target images. Instead, we learn to approximate editing operations on individual source images. We find that linear trajectories in latent space can capture simple image manipulations, e.g., zoom-vectors and shift- veetors although we also obtain similar results using nonlinear trajectories.
Dataset bias Biases from training data and network architecture both impact the generalization ca- pacity of learned models (Torralba & Efros 2011 Geirhos et al. [001]] Amiii et al.). Dataset biases partly comes from human preferences in taking photos: we tend take pictures in specific "canonical"" views that are not fully representative of the entire visual world (Mezuman & Weiss, 2012 Jahanian et al., 2015). Consequently, models trained with these datasets inherit their biases. This may result in models that misrepresent the given task such as tendencies towards texture bias rather than shape bias on ImageNet classifiers (Geirhos et al. 2018) and in turn limits their generalization performance tories transform the output corresponding to various image editing operations, but ultimately we are constrained by biases in the data and cannot extrapolate arbitrarily far beyond the data's support.
similar objectives (Azulay & Weiss 2018 Our latent space trajec-
Generative models for content creation The recent progress in generative models has opened in- teresting avenues for content creation (Brock al.) 2018 Karras al. 2018) including applications that enable users to fine-tune the generated output (Sion;; Zhu et al. 2016 Bau et al. 2018). A by-product the current work is enable users to modify image properties by turning a single knob the magnitude of the learned transformation in latent space. We further demonstrate that these image manipulations are not just a simple creativity tool: they also provide us with a window into biases and generalization capacity of these models.
Applications of latent space editing Image manipulations using generative models suggest sev- eral interesting downstream applications. For example, [Dntoon et al.] (2019) learns linear walks corresponding to various facial characteristics - they use these to measure biases facial attribute detectors, whereas we study biases in the generative model that originate from training data. [Shen] et al. (2019) also assumes linear latent space trajectories and learns paths for face attribute editing according to semantic concepts such as age and expression, thus demonstrating disentanglement of the latent space. White (2016) suggests approaches to improve the learned manipulations, such as using spherical linear interpolations, resampling images to remove biases in attribute vectors, and using data augmentation as a synthetic attribute for variational autoencoders. [Goetschalckx et al. (019) applies linear walk to achieve transformations corresponding to cognitive properties of an image such as memorability, aesthetics, and emotional valence. Unlike these works, we do not re- quire an attribute detector or assessor function to learn the latent space trajectory, and therefore our loss function is based on image similarity between source and target images. In addition to linear walks, we explore using non-linear walks parametrized by neural networks for editing operations.
3 METHOD
Generative models such as GANs (Goodfellow et al. 2014) learn mapping function G such that
G + T. Here, z is the latent code drawn from a Gaussian density and x is an output, e.g., an image. Our goal is to achieve transformations in the output space by moving in latent space, as shown in Fig.2] In general, this goal also captures the idea in equivariance, in which transformations in the input space result in equivalent transformations in the output space (c.f. Hinton et al. (011
Cohen et al. (0199) Lenc & Vedaldi (2015).
Objective We want to learn an N-dimensional vector representing the optimal path in latent space for a given transformation. The vector is multiplied with continuous parameter a which signifies the step size: large values correspond to greater degree of transformation, while small values correspond to a lesser degree. Formally, we learn the walk w by minimizing the objective function:
" = arg min E.
edit((())
(1)
w
3

---

Published as a conference paper at ICLR 2020
Latent Space
Image Space
Edit
crp
OR
Gtz aw)
edct
Linear
NonTinear
G
We aim to find path in space to transform the generated image G(z) to its edited version edi ti (G(, e.g., an ax zoom. This walk results in the generated image G(2 aw) when we choose linear walk, or G()(f(...()))) when we choose non-linear walk.
Figure 2:
Here, L measures the distance between the generated image after taking an a-Step in the latent direction G( + aw) and the target edit(G(z), a) derived from the source image G(z). We use L2 loss as our objective L. however we also obtain similar results when using the LPIPS perceptual image similarity metric (Zhang al. 2018) (see Appendix [..4.]. Note that we can learn this walk in a fully self-supervise manner we perform the edit(. operation on an arbitrary generated image and subsequently the vector to minimize the objective. Let model (a) denote the optimized transformation vector w" with the step size defined as model (a) = G(z + aww)..
The previous setup assumes linear latent space walks, but we can also learn non-linear trajectories in which the walk direction depends on the current latent space position. For the non-linear walk, we learn a function, f*),.. which corresponds to a small e-step transformation edit (G(z), e). To apply/ recursively, mimicking discrete Euler ODE approxima-
achieve bigger transformations, tions. Formally, for a fixed c, we minimize
L= E [ (Gdt((
where (-) is nth-orddr function composition ((.()....)) and f()) is parametrized with neural network. We discuss further implementation details in Appendix A.4 We use this function composition approach rather than the simpler setup of G(z + aNN(=)) because the latter learns to ignore the input when takes continuous values, and is thus equivalent to the previous linear trajectory (see Appendix A.3 for further details).
Quantifying Steerability We further seek to quantify how well we can achieve desired image ma- nipulations under each transformation. To this end, we compare the distribution of given attribute, e.g., "luminance", in the dataset versus images generated after walking in latent space.
For color transformations, we consider the effect of increasing or decreasing the a coefficient cor- responding to each color channel. To estimate the color distribution of model-generated images, we randomly sample N 100 pixels per image both before and after taking step in latent space. Then, we compute the pixel value for each channel, or the mean RGB value for luminance, and normalize the range between 0 and 1.
For zoom and shift transformations, we rely on an object detector which captures the central object in the image class. We use a MobileNet-SSD v/ (Liu et al., 2016) detector to estimate object bounding boxes, and average over image classes recognizable by the detector. For each successful detection, take the highest probability bounding box corresponding to the desired class and that to quantify the amount of transformation. For the zoom operation, we use the area of the bounding box take the center X
normalized by the area of the total image. For shift in the X and Y directions, and Y coordinates of the bounding box, and normalize by image width or height.
Truncation parameters in GANs (as used in Brock al. (2018); Karras al. (2018)) trade off be- tween the diversity of the generated images and sample quality. When comparing generated images the dataset distribution, we use the largest possible truncation for the model and perform similar cropping and resizing of the dataset as done during model training (see Brock et al. (2188). When comparing the attributes of generated distributions under different magniudes each other but not to the dataset, we reduce truncation to 0.5 to ensure better performance of the object detector.
Reducing Transformation Limits Equations I and learn a latent space walk assuming pre- trained generative model, thus keeping the model weights fixed. The previous approach allows us
4

---

Published as a conference paper at ICLR 2020
to understand the latent space organization and limitations in the model's transformation capacity. To overcome these limits, we explore adding data augmentation by editing the training images with each corresponding tramsformation, and train the generative model with this augmented dataset. We also introduce a modified objective function that jointly optimizes the generator weights and a linear walk vector:
G" w = arg min (Cedit + LGAN), G,
(3)
where the edit loss encourages low 1.2 error between learned transformation and target image:
Ledit L2 (G(=+auu) - edit(G()),
(4)
The GAN loss optimizes for discriminator error:
LGAN max (Pe.a
(5)
where we draw images x from the training datasct and perform data augmentation by applying the edit operation them. This optimization approach encourages the generator organize its latent space so that the transformations lie along linear paths, and when combined with data augmentation, results in larger transformation ranges which we demonstrate in Sec. 4.4
4 EXPERIMENTS
We demonstrate approach using BigGAN (Brock et al. 2018), class-conditional GAN trained on 1000 ImageNet categories. We learn a shared latent space walk by averaging across the image categories, and further quantify how this walk affects each class differently. We focus linear walks in latent space for the main text, and show additional results on nonlinear walks in Sec. 4.3
and Appendix B.4.2 We also conduct experiments StyleGAN (Karras et al.] 2018), which an unconditional style-based generator architecture in Sec. 4.3 and Appendix B.5
4.1
WHAT IMAGE TRANSFORMATIONS CAN WE ACHIEVE IN LATENT SPACE?
0.01
0.32 -Zoom
0.03
0.21
0.58
0.13
Shift Y
0.05
0.09
L 0.05
Rotate 2D
Figure 3: Transformation limits. As we increase the magnitude of w², the operation either does not transform the image any further, or the image becomes unrealisitic. Below each figure we also indicate the average LPIPS perceptual distance between 200 sampled image pairs of that category. Perceptual distance decreases as we move farther from the source (center image), which indicates that the images converging.
We show qualitative results of the learned transformations in Fig. By steering in the generator latent space, we learn a variety of transformations a given source image (shown in the center panel each transformation). Interestingly, several priors come into play when learning these image transformations. When shift daisy downwards in the Y direction, the model hallucinates that the sky exists on the top of the image. However, when we shift the daisy up, the model inpaints the remainder of the image with grass. When alter the brightness of image, the model transitions between nighttime and daytime. This suggests that the model can extrapolate from the original source image, and still remain consistent with the image context.
5

---

Published as a conference paper at ICLR 2020
Redness +
Brightness
-llueness
Percepttal oo
Perceptual Diezzoor
0.4
Perceptual Change
1.0-
Poreppiual
02
jollyfib
gubllinh
voloxzo
spocts
Figure 4: Each row shows how a single latent direction w" affects two different ImageNet classes. We observe that changes consistent with semantic priors (e.g.. "Volcanoes" explode, "Alps" do not). Boxplots show the LPIPS perceptual distance before and after transformation for 200 samples per class.
However, when increase the step size of a, observe that the degree to which achieve each transformation is limited. In Fig. we observe two potential failure cases: one in which the the image becomes unrealistic, and the other in which the image fails transform any further. When we try to zoom in on a Persian cat, we observe that the cat no longer increases size beyond some point, and in fact consistently undershoots the target zoom. On the other hand, when we try to zoom out on the cat, we observe that it begins to fall off the image manifold, and does not become any smaller after some point. Indeed, the perceptual distance (using LPIPS) between images decreases as we push towards the transformation limits. Similar trends hold with other transformations: we are able to shift a lorikeet up and down to some degree until the transformation yields unrealistic output, and despite adjusting on the rotation vector, we are unable to rotate pizza. Are the limitations to these transformations governed by the training dataset? In other words, are our latent space walks limited because in ImageNet photos the cats are mostly centered and taken within certain size? We seek to investigate and quantify these biases in the next sections.
An intriguing characteristic of the learned trajectory is that the amount it affects the output depends on the image class. In Fig. we investigate the impact the walk for different image categories under color transformations. By moving the direction of redness vector, fully recolor a jellyfish, but we are unable to change the color of a goldfinch, which remains yellow which slight changes in background textures. Likewise, increasing brightness changes erupting volcano to a dormant one, but does not have much effect on Alps, which only transitions between night and day. In the third example, use our latent walk turn red sports cars blue, but cannot recolor firetrucks. Again, perceptual distance over image samples confirms these qualita- 8.14,
able to success-
tive observations: a sample t-test yields < 0.001 for volcano/alp, and t = 6.84, p < 0.001 for sports car/fire engine. We hypothesize that different impact of the shared transformation on separate image classes relates to the variability in the underlying dataset. The overwhelming majority of firetrucks are red but sports cars appear in a variety of colors. Therefore, our color transformation is constrained by the dataset biases of individual classes.
20.77, p < 0.001 for jellyfishgooldinch,
With shift, we can move the distribution of the center object by varying a. In the underlying model, the center coordinate of the object is most concentrated at half of the image width and height, but transformation, the mode of the transformed distribution
after applying the shift in X and shift width/height. To quantify the distribution changes, varies between 0.3 and 0.7 of the image compute the area of intersection between the original model distribution and the distribution after applying each transformation and observe that the intersection decreases as we increase or decrease the magnitude of a. However, our transformations are limited to certain extent - if we increase
but apparently blue fire trucks do exist! DiGraziia 2019

---

Published as a conference paper at ICLR 2020
x10- Luminance
Shift
X10
Shift
X10-1
Zoom
8
1.0
3-
PDF
PDF
PDF
PDF
0.5
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.5
L0
Picel Intensity
X
Y
Area
1.0
Intersetion
Interection
Intersectoon
Infersetionn
0.5
0.5
0.5
0.5
0.0
0.01
0.0
0.0
200
200
230
0
200
-2.5
0.0 log(o)
2.6
F1D
40
40-
FID
00-
FID
20
20-
20-
20
-200
200
2000
0
200
-2.5
0.0
2.5
log (x)
Figure 5: Quantifying the extent of transformations. We compare the attributes of generated im- ages under the raw model output G(z). compared the distribution under learned transformation madel(a). We measure the intersection between G(z) and model (a), and also compute the FID) on the transformed image limit our transformations to the natural image manifold.
beyond 150 pixels for vertical shifts, we start to generate unrealistic images, as evidenced by a sharp rise in FID and converging modes in the transformed distributions (Fig. [Jolumms 2 & 3).
We perform a similar procedure for zoom, by measuring the area of the bounding box for the de- tected object under different magnitudes of a. Like shift, observe that subsequent increases in magnitude start to have smaller and smaller effects on the mode of the resulting distribution (Fig. 5 last column). Past an 8x zoom in out, we observe an increase in the FID signifying decreasing image quality. Interestingly for zoom, the FID under zooming in and zooming out is anti-symmetric,, indicating that how well we can zoom-in and retain realisitic images differs from that of zooming out. These trends are consistent with the plateau in transformation behavior that we qualitatively observe Fig. 3 Although we can arbitrarily increase the a step size, after some point we are unable to achieve further transformation and risk deviating from the natural image manifold.
4.2 How DOES THE DATA AFFECT THE TRANSFORMATIONS!"
the extent to which we can transform each class, as we observed in Fig. due to limited variability in the underlying dataset for each class? One way of quantifying this is measure the difference in transformed model means, model (+a) and model (-a), and compare it to the spread of the dataset distribution. For each class, we compute standard deviation of the dataset with respect statistic of interest (pixel RGB value for color, and bounding box area and center value for zoom and shift transformations respectively). We hypothesize that if the amount of transformation is biased depending on the image class, we will observe a correlation between the distance of the mean shifts and the standard deviation of the data distribution.
More concretely, we define the change in model means under a given transformation as:
Auto [kk.mode1(-a' (+a+)) -aataaaa"""a_aaaaaa"" Mkkaadel(-a")
for given class l and we a* to be largest and smallest values used in training. The degree to which we achieve each transformation is a function of a, so we use the same a value for all classes that is large enough to separate the means of (kkmmdell (a") and (k a") under
7

---

Published as a conference paper at ICLR 2020
Luminance
Shift X
Shift Y
Zoom
0.3
0.3
0.3
Moxel AM
0.2
Model Av
0.2
Model Ag
Model Ag
0.2
0.2
0.1
0.28
0.1
0.1
0.59
< 0.001
p < 0.001
1 0.39
p 0.001
0.37
0.001
0.0-
0.1
0.20
0.25
0.30
0.1
0.2
0.05
0.10
0.15
0.2
0.3
Data
Datu
Data
Data
Robin
0.0
1.0
Area
X-0--
Laptop
2.5
Zoom
Area
Figure 6: Understanding per-class biases. We observe correlation between the variability in the training data for ImageNet classes, and our ability to shift the distribution under latent space trans- formations. Classes with low variability (e.g., robin) limit our ability to achieve desired transfor- mations, in comparison to classes with a broad dataset distribution (e.g., laptop). To the right, we show the distribution of the zoom attribute the dataset (black) and under +a (red) and -a (green) transformations for these two examples.
transformation, but also for which the FID of the generated distribution remains below a threshold of generating reasonably realistic images (for our experiments we use T 22).
In Fig. 6 we plot the standard deviation a of the dataset on the x-axis, and the model Aμ under +a* and ax transformation on the y-axis, as defined in Eq. 6 We sample randomly from 100 classes for the color, zoom and shift transformations, and generate 200 samples of each class under the positive and negative transformations. We use the same setup of drawing samples from the model and dataset and computing the statistics for each transformation described in Sec. (.11
Indeed, we find that the width of the dataset distribution, captured by the standard deviation of random samples drawn from the dataset for each class, relates to how much we can transform. There is positive correlation between the spread the dataset and the magnitude of Aμ observed in the transformed model distributions, and the slope of all observed trends differs significantly from zero (p < 0.001 for all transformations). For the zoom transformation, we show examples of two extremes along the trend. For the "robin" class the spread in the dataset low, and subsequently, the separation Ap that we are able to achieve by applying +a" and -a" transformations limited. On the other hand, for "laptops", the dataset spread is broad; ImageNett contains images of laptops of various sizes, and we are able to attain wider shifts in the model distribution.
From these results, conclude that the amount of transformation we achieve relates to the dataset variability. Consistent with our qualitative observations in Fig. 4 we find that if the images for a particular class have adequate coverage over the entire range of given transformation, then we are better able to move the model distribution to both extremes. On the other hand, if the images for given class are less diverse, the transformation is limited by this dataset bias.
4.3 ALTERNATIVE ARCHITECTURES AND WALKS
We ran an identical set of experiments using the nonlinear walk in the BigGAN latent space (Eq 2p and obtained similar quantitative results. To summarize, the Pearson's correlation coefficient be- tween dataset and model Âμ for linear walks and nonlinear walks is shown in Table [ and full results in Appendix B.4.2, Qualitatively, we observe that while the linear trajectory undershoots the targeted level of transformation, it is able preserve more realistic-looking results (Fig. 7 The
8

---

Published as a conference paper at ICLR 2020
Zoom
ZZom
0.125v
0.25x
0.5x
1x
2xr
4x
Rx
0. 125x
0.25r
0.5x
1x
2x
4
8x
Linear
Non-linear L2
Linear Lpips
Non-linear Lpips
Figure 7: Comparison of linear and nonlinear walks for the zoom operation. The linear walk under- shoots the targeted level of transformation, but maintains more realistic output.
transformations involve a trade-off between minimizing the loss and maintaining realistic output, and we hypothesize that the linear walk functions as an implicit regularizer that corresponds well with the inherent organization of the latent space.
Luminance
Shift X
Shift Y
Zoom
Linear
0.59
0.28
0.39
0.37
Non-linear
0.49
0.49
0.55
0.60
Table 1: Pearson's 's correlation coefficient between dataset a and model Аys for measured attributes. p-value for slope < 0.001 for all transformations.
X10
modal
em
PDF
Luminance
Blueness
0.0.20
0.5-
0-0.77
0.0
0.00
0:25
0.50
0.75
1.00
Lousiamecc
Redness
Greenness
Figure 8: Distribution for luminance transformation learned from the StyleGAN cars generator, and qualitative examples of color transformations on various datasets using StyleGAN.
To test the generality of our findings across model architecture, we ran similar experiments on Style- GAN, in which the latent space is divided into two spaces, and W. As Karras et al. (2018) notes that the space is less entangled than z, we apply the linear walk to W and show results in Fig. and Appendix B.5 One interesting aspect of StyleGAN that other words, while green faces do not naturally exist in
change color while leaving
other structure in the image unchanged. the dataset, the StyleGAN model is still able generate them. This differs from the behavior of BigGAN, where changing color results in different semantics in the image, e.g., turning a dormant volcano an active one. StyleGAN, however, does not preserve the exact geometry of objects under other transformations, e.g., zoom and shift (see Appendix B.5).
4.4 TOWARDS STEERABLE GANS
So far, we have frozen the parameters of the generative model when learning latent space walk for image editing, and observe that the transformations are limited by dataset bias. Here we investigate approaches to overcome these limitations and increase model stecrability. For these experiments, we use a class-conditional DCGAN model (Radford et al. 2015) trained on MNIST digits (LeCun 1998).
To study the effect dataset biases, we train (1) a vanilla DCGAN and (2) a DCGAN with data augmentation, and then learn the optimal walk in Eq. [ affer the model has been trained refer to these two approaches in Fig. / as argmin W and argmin W + aug, respectively. We observe that adding data augmentation yields transformations that better approximate the target image and

---

Published as a conference paper at ICLR 2020
attain lower L2 error than the vanilla DCGAN (blue and orange curves in Fig. 9) Qualitatively, observe that transformations using the vanilla GAN (argmin W) become patchy and unrealistic as increase the magnitude of a, but when the model is trained with data augmentation (argmin W aug), the digits retain their structural integrity.
Rather than learning the walk vector w assuming a frozen generator, may also jointly optimize the model and linear walk parameter together, as we formalized in Eq. [3] This allows the model to learn an equivariance between linear directions in the latent space and the corresponding image transformations. We refer to this model as argmin G, W in Fig. [ Compared to the frozen gen- erator (in argmin W and argmin W + aug). the joint objective further decreases L2 error (green curve in Fig. 2 We show additional qualitative examples in Appendix B.S] The steerable range of the generator increases with joint optimization and data augmentation, which provides additional evidence that training data bias impacts the models' steerability and generalization capacity. We tried DCGAN on CIFAR10 as more complicated dataset, however were unable to get steering to be effective all three methods failed to produce realistic transformations and joint training in fact performed the worst. Finding the right steering implementation per GAN and dataset, especially for joint training, may be difficult problem and an interesting direction for future work.
Zoom
Shift X
Rotate 2D
0.3
0.5
0.5
argnin
Arguin W anyaio G.W
0.0
0.0
argmin W
v 4
V 9 G
3
3
1
1
A &
& & K
K V &
K
argmin aug
9
9
4
4
2
2
2 2
2
2
2
2
2
argmin G,W
5
5
7
6 M
G
G
C
&
&
Figure 9: Reducing the effect of transformation limits. Using DCGAN model on MNIST digits, compare the 1.2 reconstruction errors on latent space walks for models trained with vanilla GANs without (argmin W) and with data augmentation (argmin W aug). We also compare to jointly optimizing the generator and the walk parameters with data augmentation (argmin G, W), which achieves the lowest L2 error.
CONCLUSION
GANs are powerful generative models, but are they simply replicating the existing training data- points, can they to generalize beyond the training distribution? We investigate this question by exploring walks in the latent space of GANs, We optimize trajectories in latent space to reflect sim- ple image transformations in the generated output, learned in a self- the model is able to exhibit characteristics of extrapolation we are able to "steer" the generated output to simulate camera zoom, horizontal and vertical movement, camera rotations, and recol- orization. However, our ability to naively move the distribution is finite: we can transform images some degree but cannot extrapolate entirely outside the support of the training data. To increase model steerability, we add data augmentation during training and jointly optimize the model and walk trajectory. Our experiments illustrate the connection between training data bias and the result- ing distribution of generated images, and suggest methods for extending the range of images that the models are able to create.
manner. We find that
ACKNOWLEDGEMENTS
We would like to thank Quang H Le, Lore Goetschalekx, Alex Andonian, David Bau, and Jonas Wulff for helpful discussions. This work was supported by Google Faculty Research Award to P.I., and U.S. National Science Foundation Graduate Research Fellowship to L.C.
REFERENCES
Alexander Amini, Ava Soleimany, Wilko Schwarting, Sangeeta Bhatia, and Daniela Rus. Uncover- ing and mitigating algorithmic bias through learned latent structure.
10

---

Published as a conference paper at ICLR 2020
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize poorly to small image transformations? arXiv preprint arXiv: 12177,
David Bau, Jun- Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial net- works. arXiv preprini arXiv: 1..1.9911.. 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1006, 2018.
Taco S Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolu- tional networks and the icosahedral cnn. arXiv preprint arXiv:1902.04615, 2019.
Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru. Detecting bias with genera- tive counterfactual face attribute augmentation. arXiv preprint arXiv: 2019.
Bella DiGrazia. Swampscott fd debuts new blue fire truck. 2019, https: eeteemlieee
2019-09-18. http:///26.......oooooteeeeeeeen
accessed
William T. Freeman and Edward H Adelson. The design and use of steerable filters. IEEE Transac- fions on Pattern Analysis & Machine Intelligence, (9):891-906, 1991.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv:1811.1223: 2018.
Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defini- tions of cognitive image properties. arXiv preprint arXiv: 10110. 12, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In Interna- fional Conference on Artificial Neural Networks, pp. 44-51. Springer, 2011.
Ali Jahanian, SVN Vishwanathan, and Jan Allebach. Learning visual balance from large-scale datasets of aesthetically highly rated images. In Human Vision and Electronic Imaging XX, vol- ume 9394, pp. 93940Y. International Society for Optics and Photonics, 2015.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 2017.
Tero Karras, Samuli Laine, and Timo Aila. A based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.0490 044444 2018.
Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10: 1755-1758, 2009.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv: 1412.6980 6,900, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Information Processing Systems, pp. 10236-10245, 2018.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist// 1998.
Karel Lene and Andrea Vedaldi. Understanding image representations by measuring their equiv- ariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991-999, 2015.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
11

---

Published as a conference paper at ICLR 2020
Elad Mezuman and Yair Weiss. Learning about canonical views from internet image collections. In Advances in neural information processing systems, pp. 719-727, 2012.
Thomas Möllenhoff and Daniel Cremers. Flat metric minimization with applications in generative modeling. arXiv preprint (rXivv1905.047300 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint urXiv:/51..064.5 2015.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. arXiv arxp:rnnnax 2019.
Joel Simon. Ganbreeder. http://tttps:///aabreeeed
accessed 2019-03-22.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. 2011.
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7064-7073, 2017.
Tom White. Sampling generative networks. arXiv preprint www.vv:6000044400 2016.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipu- lation the natural image manifold. In European Conference on Computer Vision, pp. 597-613 Springer, 2016.
A METHOD DETAILS
A. OPTIMIZATION FOR THE LINEAR WALK
We learn the walk vector using mini-batch stochastic gradient descent with the Adam opti- mizer (Kingma & Ba 2014) in tensorflow, trained on 20000 unique samples from the latent space z. We share the vector w across all ImageNet categories for the BigGAN model.
A.2 IMPLEMENTATION DETAILS FOR LINEAR WALK
We experiment with number different transformations learned the latent space, each corre- sponding to a different walk vector. Each of these transformations can be learned without any direct supervision, simply by applying our desired edit to the source image. Furthermore, the parameter allows vary the extent of the transformation. We found that slight modification to each transformation improved the degree to which we were able to steer the output space: we scale differently for the learned transformation G(z + a,w), and the target edit eGi=(G()), We detail each transformation below:
Shift. We learn transformations corresponding shifting an image in the horizontal X direction and the vertical Y direction. We train on source images that are shifted -a, pixels to the left and a pixels the right, where we set a to be between zero and one-half of the source image width or height D. When training the walk, we enforce that the ag parameter ranges between -1 and I; thus for random shift by pixels, we use the value Og = at D. We apply a mask to the shifted image, so that we only apply the loss function the visible portion of the source image. This forces the generator to extrapolate on the obscured region of the target image.
Zoom. We learn walk which is optimized to zoom in and out up to four times the original image. For zooming in, we crop the central portion of the source image by some a amount, where 0.25 < ag < 1 and resize it back to its original size. To zoom out, downsample the image by a, where < ag < 4. To allow for both a positive and negative walk direction, we set ag = log(ar), Similar to shift, mask applied during training allows the generator inpaint the background scene.
12

---

Published as a conference paper at ICLR 2020
Color. We implement color continuous RGB slider, e.g., 3-tuple a (dig. ac, ag), where each aR, ag. ag can take values between [-0.5, 0.5] in training. To edit the source image, we simply add the corresponding a, values to each of the image channels. Our latent space walk is parameterized as z + 0kg = 2 + o + Wg + aBB8 where we jointly learn the three walk directions wn,, WG, and wg-
Rotate in 2D. Rotation in 2D is trained in a similar manner as the shift operations, where we train with 45 < Or 45 degree rotation. Using R 45, scale Og - Ot/R. R. We a mask enforce the loss only on visible regions of the target.
Rotate in 3D. We simulate 3D rotation using perspective transformation along the Z-axis, essentially treating the image as rotating billboard. Similar to the 2D rotation, we train with -45 I at I 45 degree rotation, we scale ag at/ where R 45, and apply mask during training.
A.3 LINEAR NN(=) WALK
Rather than defining w as a vector in space (Eq. D. one could define as function that takes a input and maps it to the desired z' after taking variable-sized step in latent space. In this case, we may parametrize the walk with a neural network w = NN(2), and transform the image using G(= + aNN(2)). However, function z.
show in the following proof, this idea will not learn to let w be
Proof. For simplicity, let E. [A(G(= ow)) edd,e((((()) a))] where a is an arbitrary scalar value. Note that for the target image, two equal edit operations is equivalent to performing single edit of twice the size (e.g., shifting by 10px the same as shifting by 5px twice; zooming by 4x is the same as zooming by 2x twice). That is,
F(z).
We optimize for J(w a)
edit (G(2), 2a) edittedit((()))))).))
To achieve this target, starting from an initial 2, we can take two steps of size a in latent space as follows:
= - aF 22 21 oF(21)
However, because we let a take on any scalar value during optimization, our objective function enforces that starting from z and taking a step of size 2a equals taking two steps of size a:
= +aF(22) = + aF(z1)
Therefore:
z + 2aF(2) af(22) af aF(2))
aF(z)=oFF)) oF(z)=
F(=)=F(a>)
Thus F(.) simply becomes a linear trajectory that is independent of the input z.
A.4 OPTIMIZATION FOR THE NON-LINEAR WALK
Given the limitations of the previous walk, we define our nonlinear walk F(2) using discrete step sizes e. We define F(z) tion, rather than variable a step. We then renormalize the magnitude z. This approach mimics the Euler method for solving ODEs with discrete step size, where assume that the gradient of the transformation in latent space of the form de The key difference from A.3 is the fixed step size, which avoids optimizing for the equality in 7
NN(2), where the neural network NN learns a fixed step transforma-
de NN(z) and we approximate 2++1 = 2, +
We use a two-layer neural network to parametrize the walk, and optimize over 20000 samples using the Adam optimizer before. Positive and negative transformation directions are handled with two neural networks having identical architecture but independent weights. We set e to achieve the same transformation ranges the linear trajectory within 4-5 steps.
13

---

Published as a conference paper at ICLR 2020
B ADDITIONAL EXPERIMENTS
B.I MODEL AND DATA DISTRIBUTIONS
How well does the model distribution of each property match the dataset distribution? the gener- ated images do not form good approximation of the dataset variability, we expect that this would also impact our ability to transform generated images. In Fig. I0 we show the attribute distributions of the BigGAN model G(z) compared to samples from the ImageNet dataset. We show corre- sponding results for StyleGAN and its respective datasets in Appendix B.5 While there is some bias in how well model-generated images approximate the dataset distribution, hypothesize that additional biases in our transformations come from variability in the training data.
B.2 QUANTIFYING TRA LIMITS
We observe that when we increase the transformation magnitude a in latent space, the generated images become unrealistic and the transformation ceases have further effect. We show this quali- tatively in Fig. 3 To quantitatively verify this trends, we can compute the LPIPS perceptual distance of images generated using consecutive pairs of a; and 04+1. For shift and zoom transformations, perceptual distance is larger when (or log(a) for zoom) is near zero, and decreases as the the magnitude of a increases, which indicates that large a magnitudes have smaller transformation ef- fect, and the transformed images appear more similar. On the other hand, color and rotate in 2D/3D exhibit a steady transformation rate as the magnitude of a increases.
Note that this analysis does not tell how well we achieve the specific transformation, nor whether the latent trajectory deviates from natural-looking images. Rather, it tells us how much we manage to change the image, regardless of the transformation target. To quantify how well each transformation achieved, we rely on attribute detectors such as object bounding boxes (see B.3
B.3 DETECTED BOUNDING BOXES
To quantify the degree to which we are able to achieve the zoom and shift transformations, we rely on pre-trained MobileNeet-SDD v object detection model. In Fig. 12 and 13 we show the results of applying the object detection model to images from the dataset, and images generated by the model under the zoom, horizontal shift, and vertical shift transformations for randomly selected values of a, to qualitatively verify that the object detection boundaries are reasonable. Not all ImageNet images contain recognizable objects, we only ImageNet classes containing objects recognizable by the detector for this analysis.
B.4 ALTERNATIVE WALKS BIGGAN
B.4.1 LPIPS OBJECTIVE
In the main text, we learn the latent space walk w by minimizing the objective function:
J(w, a.) = E_ [C(G(z aw), edi edi)
(8)
using Euclidean loss for L. In Fig. 14 we show qualitative results using the LPIPS perceptual similarity metric (Zhang et al. 2018) instead Fuclidean loss. Walks were trained using the same parameters as those in the linear-1.2 walk shown in the main text: we use 20k samples for training,
with Adam optimizer and learning rate 0.001 for zoom and color, 0.0001 for the remaining edit operations (due to scaling of a).
B.4.2 NON-LINEAR WALKS
Following B.4.2
modify objective to use discrete step sizes rather than continuous steps. We learn function F(z) to perform this e-step transformation on given latent code z, where F(z) parametrized with neural network. We show qualitative results in Fig. 15 We perform the same
set of experiments shown the main text using this nonlinear walk Fig. T6 These experiments
14

---

Published as a conference paper at ICLR 2020
exhibit similar trends observed in the main text able to modify the generated distribu- tion of images using latent space walks, and the amount to which we can transform related to the variability in the dataset. However, there greater increases in FID when apply the non-linear transformation, suggesting that these generated images deviate more from natural images and look less realistic.
B.4.3 ADDITIONAL QUALITATIVE EXAMPLES
We show qualitative examples for randomly generated categories for BigGAN L2near-L2, linear LPIPS, and nonlinear trajectories in Figs. 17 18 19 respectively.
B.5 WALKS IN STYLEGAN
We perform similar experiments for linear latent space walks using StyleGAN models trained on the LSUN cat, LSUN car, and FFHQ face datasets. As suggested by Karras et al. (2018), we learn the walk vector in the intermediate W latent space due improved attribute disentanglement in W.
We show qualitative results for color, shift, and zoom transformations in Figs. 20 22 24 and corre-
sponding quantitative analyses Figs. 21 23
We show qualitative examples for the comparison
of optimizing in the W and latent spaces in Stylegan in 28
B.6 WALKS IN PROGRESSIVE GAN
We also experiment with the linear walk objective in the latent space of Progressive GAN Karras al. (2017). One interesting property of the Progressive GAN interpolations is that they take much longer to train to have visual effect - for example for color, we could obtain drastic color changes in Stylegan W latent space using as few as 2k samples, but with progressive gan, we used 60k samples and still did not obtain strong of an effect. This points to the Stylegan w latent space being more "flexible" and generalizable for transformation, compared to the latent space of progressive GAN. Moreover, qualitatively observe some entanglement in the progressive gan transformations for example, changing the level zoom also changes the lighting. We did not observe big effects in the horizontal and vertical shift transformations. Qualitative examples and quantitative results are shown in Figs. 26 27
B.7 QUALITATIVE EXAMPLES FOR ADDITIONAL TRANSFORMATION
Since the color transformation operates individual pixels, we can optimize the walk using seg- mented target- for example when learning walk for cars, we only modify pixels in segmented car region when generating edit (G(2), a). StyleGAN is able to roughly localize the color transforma- tion to this region, suggesting disentanglement of different objects within the W latent space (Fig. 29 left) as also noted in Karras et al. 2018); [hhnn al. (2019). We also show qualitative results adjust image contrast (Fig. 29 right), and for combining zoom, shift X, and shift Y transformations (Fig. 30
B.8 ADDITIONAL RESULTS FOR IMPROVING MODEL STEERABILITY
We further test the hypothesis that dataset variability impacts the amount by comparing DCGAN models trained with and without data augmentation. Namely, with data augmentation, the discriminator is able to see edited versions of the real images. We also jointly train the model and the walk trajectory which encourages the model to learn linear walks. For zoom, horizontal shift, and 2D rotate transformations, additional samples for three training approaches without data augmentation, with data augmentation, and joint optimization appear in Fig. 31 33 Qualitatively, transformations using the model trained without data augmentation degrade the digit structure as magnitude increases, and may change one digit to another. Training with data augmentation and joint optimization better preserves digit structure and identity.
able transform
15

---

Published as a conference paper at ICLR 2020
Luminance
x:0
Shift
Shift Y
x10
Zoom
211
data
1.5
model
modil
1.0-
10000
0.0
0.0
0.2
Plw. Esbensity
0.4
1.0
112
18.8
Conter
0.0
0.5
3.0
0.2
0.4
Cetter
0.6
11.00
Figure 10: Comparing model versus dataset distribution. We plot statistics of the generated under the color (luminance), zoom (object bounding box size), and shift operations (bounding box center), and compare them to the statistics of images in the training dataset.
Luminance
Shift X
Shift Y
Dattees
0.4-
0.4
80
0.3
60
233
000
260
20
-301
Zoom
Rotate 2D
Rotate 3D
0.5
Papoptual Dataue
Peptital Datuure
0.0
6.4
0.4-
Figure 11: LPIPS Perceptual distances between images generated from pairs of consecutive a; and a,+1- We sample 1000 images from randomly selected categories using BigGAN, transform them according the learned linear trajectory for each transformation. We plot the mean perceptual distance and one standard deviation across the 1000 samples (shaded area), well as 20 individual samples (scatterplot). Because the Rotate 3D operation undershoots the targeted transformation, we observe more visible effects when we increase the magnitude.
Figure 12: Bounding boxes for random selected classes using ImageNet training images.
16

---

Published as a conference paper at ICLR 2020
Figure 13: Bounding boxes for random selected classes using model godel-generated images for zoom and horizontal and vertical shift transformations under random values of a.
Figure 14: Linear walks in BigGAN, trained to minimize LPIPS loss. For comparison, we show the same samples as Fig. (which used linear walk with L2 loss).
Figure 15: Nonlinear walks in BigGAN, trained to minimize L2 loss for color and LPIPS loss for the remaining transformations. For comparison, we show the same samples in Fig. | (hich used linear walk with L2 loss), replacing the linear walk vector w with a nonlinear walk.
17

---

Published as a conference paper at ICLR 2020
Luminance
Shift X
Shift Y
Zoom
1.5
LG
0.0
0.2
0.6
0.5
0.6
0.
1.0
06
0%
Ocalee
1.00
1.00
1.20
1.00
0.
0.30
0.50
0.55
0:25-
0120
0.00
kogle)
00-
50-
40-
0
-0--
30-
10
20-
20
-1.0
-1.5
0.0
0.5
-20
100
100)
200
-00
100
100
300
-2
kodlo)
0.5
0.4
0.1
0.4
0.3
Model
0.3
Model
0.2
Model
Model
0.2
0.2
p=0.49
p=0.49
.p=0..5
p=0.60
0.20
0.25
0.30
0.10
0.15
0.20
0.05
0.10
0.15
0.2
0.3
Data
Data
Data
Data
Figure 16: Quantitative experiments for nonlinear walks in BigGAN. We show the attributes of generated images under the raw model output G(z). compared to the distribution under learned transformation mode la)) the intersection area between G(z) and model (a), FID score on trans- formed images, and scatterplots relating dataset variability the extent model transformation.
18

---

Published as a conference paper at ICLR 2020
Zoom +
Color
Shift X +
Shift Y
Rotate
Rotate 3D
Figure 17: Qualitative examples for randomly selected categories in BigGAN, using the linear tra- jectory and L2 objective.
19

---

Published as a conference paper at ICLR 2020
Zoom
Color +
Shiftx +
Shift
Rotate 2D
Rotate 3D
Figure 18: Qualitative examples for randomly selected categories BigGAN, using the linear tra- jectory and LPIPS objective.
20

---

Published as a conference paper at ICLR 2020
Zoom +
Color
Shift X
Shift Y +
Rotate 2D
Rotate 3D
Figure 19: Qualitative examples for randomly selected categories BigGAN, using a nonlinear trajectory.
21

---

Published as a conference paper at ICLR 2020
Zoom +
Color
Shift
Shift Y
Rotate 2D
Rotate 3D
Figure 20: Qualitative examples for learned transformations using the StyleGAN car generator.
22

---

Published as a conference paper at ICLR 2020
1.5
10
Luminance
1.00
X30-3
Shift X
Shift Y
Zoom
1.0
0.95
0.00
0.001
01:55
11:01
100
100
0.35
0.30
0.75
0.75
1.00
0.50
0.10
Leauaaae
Sui
S5-BY
Zoem
1.0
1.0
1.0-
1.0-
Infonecion
Infenection
Infenecton
Interestioo
0.5
0.5-
0.0
0.0
0.0
0.0
koglo)
75
200
FID
40-
FID
50
FID
20-
25
1000
100
100
100
log(a:
x x0
100
1.0
1.0
date model
1.3-
PDF
0.5
data model
dhde musil
PDF
dasa model
6.0
0.0
StanX
0.5
0.0
0.5 Shift
0.0
0.5
1.0
Figure 21: Quantitative experiments for learned transformations using the StyleGAN generator.
23

---

Published as a conference paper at ICLR 2020
Zoom
Color
Shift
Shift Y
Rotate 2D
Rotate 3D)
Figure Qualitative examples for learned transformations using the StyleGAN cat generator.
24

---

Published as a conference paper at ICLR 2020
x10+2
Luminance
Shift X
200-2
Shift
Zoom
2.0
1.0
PDP
PDP
0.0
0.50
0.35
0.50
0.10
1.00
0.00
0.25
0.50 SainX
0.70
1.00
0.25
0.75
SaiRt
-25
220
184
meat
Intersetion
Interellion
0.5-
0.5-
0.5-
0.5
100
100
100
log(o))
40
FID
30
20
50
20
20
100
100
1000
0
100
log(os
-100-
x10
x20-2
X10
mode
1.5
PDP
PDP
data model
PDP
PDF
1.0-
data modde
date
0.5
30000
Laninance
ShilIX
Shitl
Figure 23: Quantitative experiments for learned transformations using the StyleGAN cat generator.
25

---

Published as a conference paper at ICLR 2020
Zoom
Color +
Shift
Shift
Rotate 2D
Rotate 3D
Figure 24: Qualitative examples for learned transformations using the StyleGAN FFHQ face gener- ator.
26

---

Published as a conference paper at ICLR 2020
Luminance
Shift X
XI0
Shift
Zoom
2.0
1.0
1122
0.0)
0.5
0.00
0.25
01241
0.70
L(00
0.6
0.6
04
100
08.
0.00
0.25
0.255
1.00
Lestissttann
StaY
2008
and
1.0
1.04
1.0
1.0
Intevettion
Intoraction
Intersetion
Infersetiee
6.5-
6.0
100
100
1000
1001
koglee)
150
100
100
FID
FID
FID
200
100
100
1.00
100
100
100
logfa:
10
X010
10
date
deta
data
model
modeel
1.01
modd
PDF
2-
data modid
0.00
0.0
0.5
0.4
0.6
0.8
0.6
0.8
0.0
0.5
StaRX
Zoom
Figure 25: Quantitative experiments for learned transformations using the StyleGAN FFHQ face generator. For the values in which no face is detected. We use the dlib face detector (King 2009) for bounding box coordinates.
zoom operation not all faces are detectable; we plot the distribution as zeros for
27

---

Published as a conference paper at ICLR 2020
Zoom
Color
Shift X
Shift Y
Figure 26 Qualitative examples for learned transformations using the Progressive GAN CelebaA- HQ face generator.
Luminance
<10
Shift X
Shift Y
X10-5
Zoom
0.000
0.0
0.00
0.50 Zcom
0.55
Sun
KT5
2004
Inlonection
Intewetinn
0.5
6.54
273
200
230
-1
25
90-
FID
20
200
log(n)
daa mode
dista model
data
PDF
VDF
0.5
PDP
PDP
2-
data
Laminance
SluftY
Zoom
Figure 27:
Quantitative experiments for learned transformations using the Progressive GAN CelebA-HE HQ face generator.
28

---

Published as a conference paper at ICLR 2020
Target
latent
z latent
Brightness
Brightness
Figure 28: Comparison of optimizing for color transformations in the Stylegan w and z latent spaces.
Color
Contrast
Figure 29: Qualitative examples of optimizing for a color walk with a segmented target using Style- GAN in left column and contrast walk for both BigGAN and StyleGAN in the right column.
Figure 30: Qualitative examples of a linear walk combining the zoom, shift X, and shift Y transfor- mations. First row shows the target image, second row shows the result of learning a walk for the three transformations jointly, and the third row shows results for combining the separately trained walks. Green vertical line denotes image center.
29

---

Published as a conference paper at ICLR 2020
argmin W
argmin G,W
argmin W + aug
2
3
3
3
3
3
3
Uh
U
G
6
7
7
8
&
8
8
8
8
8
8
8
8
&
Z
M
&
8
6
8
8
9
4
Figure 31: Quantitative experiments steerability with MNIST DCGAN for the Zoom transfor- mation. Odd rows are the target images and even rows are the learned transformations.
argmin W
argmin G, W
argmin W + aug
C
C
D
0
O
O
O
O
O
O
C
6
+
C
6
0
0
O
D
O
C
C
O
0
O
/
/
/
/
/
1
|
I
I
1
7
/
f
f
8 2 2 3 3 4 4
2
!
/
I
F
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2 2
2
77
_
2
2
9
2
.2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3 3
3
3
3
3
3
3
7
1
1
3
3
3
3
3
3
3
3
3
2
3
3
3
4
4
4/
4
4
4
4
4/
Y
4
Y
4
4
F
3
'''
4
4
4
4
4
4
4
7
Y
4
7
4
4
5
J
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
3
3
5
5
5
5
5
5
5
6
6
9
6
6
399
2
2
6
6
9
9
6
7
7
<-
6
6
6
7
3
3
4
6
7
7
2
2
2
3
4
7
4
7
7
7
7
7
7
7
7 7 7
7
7
4
3
7
7
7
7
7
7
7
7
7
7
9
9
7
5
7
7
7
Y
7
7
Y
8
3
8
8
8
7
7
7
E
7
7
y
8
f
r 9
J
8
&
9
9
9
9
9 9
9
7
9
9
9
9
9
9
9
2
9 9
2
9
9
9
9
9
4
4
9
9
9
9
9
Figure 32:
Quantitative experiments on steerability with MNIST DCGAN for the Shift X trans- formation. Odd rows are the target images and even rows are the learned transformations.
30

---

Published as a conference paper at ICLR 2020
argmin W
argmin G,W
argmin W + aug
O
C
/
2
2
2
2
2
2
X
2
2
2
2
2
2
2
2
2 2
R
2
L
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
2 4 4 5 4 6
J
3
3
3
3
3
&
5
4
4
4
4
4
4
of
of
4 5 1
Y
Y
5
5
5
5
5
5
9
9
S
S
5
5
5
5
5
9
6
9
6 &
6
7
7
9
% 1
7
7
7
7
7
1
7
7
7
7
1
7
7
6
&
&
&
&
X
&
Y
6
6
8
8
8
8
6
6
&
3
%
%
%
R
A
R
g
E
KG
8
E
O
O
8
8
8
9
9
4
4
4
7
7
7
9
9
9
9
7
4 4 4
7
9
9
Figure 33: Quantitative experiments steerability with an MNIST DCGAN for the Rotate 2D transformation. Odd rows are the target images and even rows are the learned transformations.
31


Reviews:
[{"review_id": "rkxJexgWqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose to study the generalization properties of GANs through interpolation. They first propose to learn a linear (and non-linear) interpolation in the latent space for a specific type of image transformation for example zoom, translation, rotation, luminance, etc... They show that linear interpolation in GANs can produce really realistic images along the path and enable to control and transform generated images to some extent. They then propose to measure to what extent the generated images can be transformed without \"breaking\".  Finally they show that the quality of the interpolation can be improved by learning the interpolation and generator jointly.\n\nI'm in favour of accepting this paper. The paper is well written and organized. The experiments and observations are very interesting and really illustrate the generalization capacity of GANs.\n\nMain argument:\n- I think those observations are very valuable to the community and are a good way to get insight into the capabilities of GANs. This also give interesting informations about the different bias present and learnt in the dataset. This could also lead to very nice applications.\n- The interpolation with StyleGAN and BigGAN seem to give qualitatively very different results. It would have been very interesting to study the quality of interpolations on more models and datasets, and compare their generalization capabilities as well as the bias present in the different datasets.\n- Does training the generator and interpolation jointly improve the quality of the generator in general ? It would have been nice to run this method on more complicated dataset like CIFAR10 and see if this method increase the overall FID score.\n\n\nMinor comments:\n- In appendix A.2 the authors explain how the range of $\\alpha$ is set for the different experiments. However it's not clear how is this range used in practice ? Do you sample uniformly $\\alpha$ in this range to train the linear interpolation ? Also how many steps are required to learn the linear interpolation ? How much the does it influence the quality of the interpolation ?\n- There is a typo in equation 6\n- In figure 6: What does the right figure represent ? especially what are the different colours ?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SylrPNUFFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work explores the extent to which the natural image manifold is captured by generative adversarial networks (GANs) by performing walks in the latent space of pretrained models. To perform these walks, a transformation vector is learned by minimizing the distance between transformed images and the corresponding images generated from transformed latent vectors. It is found that when traversing the latent space of the GAN along the direction of the transformation vector, that the corresponding generated images initially exhibit the desired transform (such as zooming or changing X position), but soon reach a limit where further changes in the latent vector do not result in changes to the image. It is observed that this behaviour is likely due to bias in the dataset which the GAN is trained on, and that by exploring the limits of the generator, biases which exist in the original dataset can be revealed. In order to increase the extents to which images can be transformed, it is shown that GANs can be trained with an augmented dataset and using a loss function that encourages transformations to lie along linear paths.\n\nOverall, I would tend towards accepting this paper. Improving the amount of control that we have over generative models is desirable for image synthesis, and this paper does a great job of demonstrating the extent to which these models can be manipulated in terms of mimicking basic transforms. Figures are very clean and informative, and experimental results are extensive. I don't have much else to say about this paper, as I did not find anything in it that concerned me, and the paper answered all of my questions.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1gLu2Q1iS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores and experiments on extrapolating attributes of images produced by GANs by manipulating their representations in latent space. Attribute manipulation is done by predicting latent space walks (linear or non-linear) and is learned in a self-supervised way by using augmented outputs of a pretrained GAN as target images.\nAuthors experimentally show dependence of range of possible attribute manipulations on the diversity of the dataset in terms of that attribute as well as propose techniques to improve it.\n\nSuggested concepts are explained in a clear way with extensive experiments confirming the findings. Techniques proposed for improving \"steerability\" of GANs are backed up by both qualitative and quantitative analysis, although missing experiments on more sophisticated datasets than MNIST.\nOverall,  I recommend to accept this paper.\n\nSeveral questions I would like the authors to address to make some details more clear and the paper more complete:\n1. Why the color distribution of generated images is evaluated on a sampled subset of pixels, not full images? (\"Quantifying steerability\" section.)\n2. On Figure 6, which classes are outlying on transformation limitation / data variability plots (bottom-right corner) and how it may be explained?\n3. While StyleGAN can not preserve geometry of objects for shift in location-based attributes, when walks are learned in the W space, have you experimented on manipulating those attributes with z space? What are the results?\n\nOther minor flaws include\n1. Pictures in Fig. 2 are mixed up between G(z) and G(z + \\alpha w)\n2. In Fig. 2 edit(G(z, \\alpha)) -> edit(G(z), \\alpha))\n3. In eq. (2) f^n(z) -> G(f^n(z))\n4. In eq. (6) +\\alpha^* -> -\\alpha^* ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers agree that the paper provide an interesting study on the ability of generative adversarial networks to model geometric transformations and a simple practical approach to how such ability can be improved. Acceptance as a poster is recommended.
Decision: Accept (Poster)