GAN "Steerability" without optimization 
Authors: Nurit Spingarn
Ron Banner
Tomer Michaeli
Venue: ICLR 2021 Spotlight (2021)
Keywords: Generative Adversarial Network
semantic directions in latent space
nonlinear walk

Abstract: Recent research has shown remarkable success in revealing "steering" directions in the latent spaces of pre-trained GANs. These directions correspond to semantically meaningful image transformations (e.g., shift, zoom, color manipulations), and have the same interpretable effect across all categories that the GAN can generate. Some methods focus on user-specified transformations, while others discover transformations in an unsupervised manner. However, all existing techniques rely on an optimization procedure to expose those directions, and offer no control over the degree of allowed interaction between different transformations. In this paper, we show that "steering" trajectories can be computed in closed form directly from the generator's weights without any form of training or optimization. This applies to user-prescribed geometric transformations, as well as to unsupervised discovery of more complex effects. Our approach allows determining both linear and nonlinear trajectories, and has many advantages over previous methods. In particular, we can control whether one transformation is allowed to come on the expense of another (e.g., zoom-in with or without allowing translation to keep the object centered). Moreover, we can determine the natural end-point of the trajectory, which corresponds to the largest extent to which a transformation can be applied without incurring degradation. Finally, we show how transferring attributes between images can be achieved without optimization, even across different categories.


Published as a conference paper at ICLR 2021
GAN " "STEERABILITY" WITHOUT OPTIMIZATION
Nurit Spingarn Eliezer
Ron Banner
Tomer Michaeli
Technion Israel Institute of Technology Habana Labs Intel f {nurits campus, tomer. momerrm@ee)) mmee) www.neee.hhh.cc.ccec ac. i i1, ron
ABSTRACT
Recent research has shown remarkable success in revealing "steering" directions in the latent spaces of pre-trained GANs. These directions correspond to seman- tically meaningful image transformations (e.g., shift, zoom, color manipulations) and have similar interpretable effects across all categories that the GAN can gener- ate. Some methods focus user-specified transformations, while others discover transformations in an unsupervised manner. However, all existing techniques rely optimization procedure to expose those directions, and offer no control over the degree of allowed interaction between different transformations. In this paper, we show that "steering" trajectories be computed in closed form directly from the generator's 's weights without any form of training or optimization. This applies user-prescriber geometrie transformations, as well to unsupervised discovery of more complex effects. Our approach allows determining both linear and non- linear trajectories, and has many advantages over previous methods. particular, we can control whether one transformation is allowed to come the expense of another (e.g. zoom-in with or without allowing translation to keep the object cen- tered). Moreover, can determine the natural end-point of the trajectory, which corresponds to the largest extent to which a transformation can be applied with- out incurring degradation. Finally, show how transferring attributes between images can be achieved without optimization, even across different categories.
INTRODUCTION
Since their introduction by Goodfellow et al. 2014), generative adversarial networks (GANs) have seen remarkable progress, with current models capable of generating samples of very high qual- ity (Brock et al.) 2018 Karras et al. 2019a 2018 2019b). In recent years, particular effort has been invested in constructing controllable models, which allow manipulating attributes of the generated images. These range from disentangled models for controlling e.g., the hair color gender of facial images (Krrras et al., 2019a Choi et al., 2018). to models that even allow specifying object rela- tions (Ashual & Wolf 2019). Most recently, it has been demonstrated that GANs trained without explicitly enforcing disentanglement, can also be easily "steered" (Jahanian et al.) 2020; Pluueeaaul]] al. 2020). These methods can determine semantically meaningful linear directions in the latent space of a pre-trained GAN, which correspond to various diffcrent image transformations, such as zoom, horizontal/vertical shift, in-plane rotation, brightness, redness, blueness, etc. Interestingly, a walk in the revealed directions typically has similar effect across all object categories that the GAN can generate, from animals to man-men made objects.
To detect such latent-space directions, the methods of Jahanian et al. (2020) and Plumerault et al. (2200) require a training procedure that limits them to transformations for which synthetic images can be produced for supervision (e.g., shift or zoom). Other works have recently presented unsu- pervised techniques for exposing meaningful directions (Voynov & Babenko 2020) Härkönen et al.) 2020 Peebles et al., 2020). These methods can go beyond simple user-specified transformations, but also require optimization or training of some sort (e.g., drawing random samples in latent space).
In this paper, we show that for most popular generator architectures, it is possible to determine meaningful latent space trajectories directly from the generator's weights without performing any kind of training optimization. As illustrated in Fig. I our approach supports both simple user- defined geometric transfornations, such as shift and zoom, and unsupervised exploration of direc- tions that typically reveals more complex controls, like the 3D pose of the camera or the blur of the

---

Published as a conference paper at ICLR 2021
User- Diser-prescriber geometric transformations
Color Transfer Original
Turget
Shin
Transferred
ShiR
Automatically detected transformations
Pose Transfer
Temperatrrs
Original
Turget
Degeh
Fledd
Tranffereel
Baddground Removal
Figure Steerability without optimization. We determine meaningful trajectories in the latent space of pre-trained GAN without using optimization. We accommodate both user-prescribed geometric transformations, and automatic detection of semantic directions. We also achieve attribute transfer without any training. All images were generated with BigGAN (Brock et al., 2018).
background. We also discuss how to achieve attribute transfer between images, even across object categories (see Fig. [ again without any training. We illustrate results mainly BigGAN, which is class-conditional, but our trajectories are class-agnostic. Our approach is advantageous over existing methods in several respects. First, it 10 x faster. Second, it seems to detect more semantic directions than other methods. And third, it allows explicitly accounting for dataset biases.
First order dataset biases As pointed out by Jahanian et al. (020), dataset biases affect the extent to which pre-trained generator can accommodate different transformations, For example, all objects in the training set centered, then no walk in latent space typically allows shifting an object too much without incurring degradation. This implies that a "steering" latent-space trajectory should have end point. Our nonlinear trajectories indeed possess such convergence points, which correspond to the maximally-transe transformndd versions of the images at the beginning the trajectories. Conveniently, the end-point be computed in closed form, that can directly jump to the maximally-transformme image without performing a gradual walk.
Second order dataset biases For example, in many datasets zoomed-out objects can appear anywhere within the image, while zoomed-in in objects are always centered. In this case, trying to apply zoom transformation may also result an undesired shift as center the enlarged object. Our unsupervised method allows controlling the extent to which transformation A comes on the expense of transformation B.
Dataset biases can also lead to coupling between transformations.
1.1 RELATED WORK
Walks in latent space
Many works use walks in GAN's latent space to achieve various effects
(e.g., (Shen et al. 2020; Radford et al. 2015 Karras et al. 2018 2019b Denton et al.) 2019 Xiao
et al.) 2018 Goetschalekx et al. 2019)). The recent works of Jahanian al. (2020) and Plumer- ault et al. (2020) specifically focus on determining trajectories which lead to simple user-spedified transformations, by employing optimization through the (pre-trained) generator. Voynov & Babenko
2

---

Published as a conference paper at ICLR 2021
P
Procan
FC2
Block
Block
Paninx
E
Block N
Panaa
Pre-trained genoralor
Figure 2: User-prescribed spatial manipulations. We calculate directions latent space whose effect the tensor at the output of the first layer, is similar to applying transformation P on that tensor. This results in the generated image experiencing the same transformation.
2020) proposed an unsupervised approach for revealing dominant directions in latent space. This technique reveals more complex transformations, such as background blur and background removal, yet also relies on optimization. Most recently, the work of Härkööen et al. 2020 studied unsu- pervised discovery of meaningful directions by using PCA on deep features of the generator. The method seeks linear directions in latent space that best map to those deep PCA vectors, and results in of non-orthogonal directions. Similarly to the other methods, it also requires very demanding training procedure (drawing random latent codes and regressing the latent directions), which can take day for models like BigGAN.
Nonlinear walks in latent space Linear latent-space trajectories may arrive at regions where the probability density is low. To avoid this, some methods proposed replace the popular Gaussian latent space distribution by other priors (Kilcher et al.j 2018), or to optimize the generator together with the latent space (Bojanowski et al. 2018). Others suggested use nonlinear walks in latent space that avoid low-probability regions. For example, Jahanian et al. (020)) explored nonlinear trajectories parametrized by two-layer neural networks, White 2016) proposed spherical paths for interpolating between two latent codes.
Hierarchical GAN architectures Recently there is tendency towards hierarchical GAN architec- tures (Krraas et al.) 2018 [019a] Brocc et al.) 2018 Choi et al. 2018). which are capable of pro- ducing high resolution images at very high quality.
known that the earlier scales such models are responsible for generating the global composition of the image, while the deeper scales are re- sponsible for more local attributes (Karras et al. 2019a Yang et al.) 2019 Härkönen et al.) 2020). Here, distil this common knowledge and show how meaningful directions be detected in each level, and how these architectures allow transferring attributes between images.
2 -SERISPED GEOMETRIC TRANSFORMATIONS
Most modern generator architectures map latent code vector = Rd having no notion of spatial coordinates, into a two-dimensional output image. In some cases (e.g., BigGAN), different parts of are processed differently. In others (e.g., BigGAN-deep), is processed as a whole. However, in all cases, the first layer maps (or part of into tensor with low spatial resolution (e.g., 4 x 4 x 1536 BigGAN 128). This tensor is then processed by a sequence of convolutional layers that gradually increase its spatial resolution (using fractional strides), until reaching the final image dimensions.
Our key observation is that since the output of the first layer already has spatial coordinates, this layer has an important role in determining the coarse structure of the generated image. This suggests that we were to apply geometric transformation, like zoom or shift, on the output of the first layer, then we would obtain a similar effect applying it directly fact, it may even allow slight semantic changes to take place due to the deeper layers that follow, which can compensate for the inability of the generator to generate the precise desired transformed image. As we now show, this observation can be used to find latent space directions corresponding simple geometric transformations.
the generated image (Fig. 2). In

---

Published as a conference paper at ICLR 2021
2.1 LINEAR TRAJECTORIES
Let us start with linear trajectories. Given a pre-trained generator G and some transformation T. our goal is to find a direction q in latent space such that G(z q) ~ [[G(2) for every z. To this end, we define P to be the matrix corresponding to T in the resolution of the first layer's output. Denoting the weights and biases of the first layer by W and b. respectively, our goal therefore to bring (= + q) + as close as possible to P(W x + b). To guarantee that this holds on average over random draws of 2, we formulate our problem as
[|
+ P(W= + bi)
min EzwpA
where Ps is the probability density function of 2, and D is a diagonal matrix that can be used to assign different weights to different elements of the tensors. For example, if P corresponds horizontal shift of one element to the right, then we would not like to penalize for differences in the leftmost column of the shifted feature maps (see Fig. 2 In this case, we set the corresponding
diagonal elements of D to 0 and the rest to 1. Assuming E z = 0, as is the case in most frameworks, the objective in I simplifies to
P)W [2] + W (I- (W
Ez-Ps
where I is the identity matrix. The first term in 2 is independent of q, and the second term is quadratic q and is minimized by
q ( w) WPD
(3)
We have thus obtained a closed form expression for the optimal linear direction corresponding to transformation P in terms of only the weights W and b of the first layer.
Figure 2 iillustrates this framework in the context of the BigGAN model, in which the feature maps at the output of the first layer are x 4. For translation, use matrix P that shifts the tensor by one element (aiming at translating the output image by one fourth its size). For zoom-in, we use a matrix P that performs nearest-neighbor 2x up-sampling, and for zoom-out we use sub-sampling by 2x. For each such transformation, we can control the extent of the effect by multiplying the steering vector q by some a O.
Figure I (top-left) and Fig. ([a) show example results for zoom and shift with the BigGAN gener- ator. As can be seen, this simple approach manages produce pronounced effects, although not using optimization through the generator, as in (Jahanian et al. 2020). Following (Jahanian et al. 2020), we use an object detector to quantify our zoom and shift transformations. Figure shows the distributions of areas and centers of object bounding boxes in the transformed images. As can be seen, our trajectories lead to similar effects to those of Jahanian et al. (2020), despite being 10 faster to compute (see Tab. TT. Please refer to App. A.T for details about the evaluation, and see additional results with BigGAN and with the DCGAN architecture of (Miyato et al. 2018 in App. A.3.
2.2
ACCOUNTING FOR FIRST-ORDER DATASET BIASES VIA NEUMANN TRAJECTORIES
With linear trajectories, the generated image inevitably becomes improbable after many steps, as (z(z + +aq) is necessarily small for large This causes the generated image to distort until even- tually becoming meaningless after many steps. One way to remedy this, is by using nonlinear trajectories that have endpoints. Here, we focus on walks in latent space, having the form
Za+1 M + g.
(4)
for some matrix M and vector q. We coin these Neumann trajectories, since unfolding the iterations leads to a Neumann series. An important feature of such walks is that if the spectral norm of M is strictly smaller than 1 (a condition we find to be satisfied in practice for the optimal M), then they have a convergence point. We use a diagonal M, which we find gives the best results. To determine the optimal M and q for a transformation P. we modify Problem (I) into
p [W q) + P00-- + bi)
min E Eg-pA M.q
(5)
Forr architectures like BigGAN, in which the first FC layer operates on a subset of the entries of the latent vector, we use refer this subset rather than the whole vector.
4

---

Published as a conference paper at ICLR 2021
Zoom
Zoom
Oua
Our Stea: Namraan Grese
Our
Siess
Shift Y
Shift Y
Our Stoer: Nearnenn Greek
Steer:
Shift X
Shift
Steek:
(a) Linear LinearTrajeee Trajectories
(b) Nonlinear NoniinearTaaeecootsss
Figure 3: Walks corresponding to geometric transformations. We compare our zoom and shift trajectories to those of the GAN steerability work (Jahanian et al. 2020). For linear paths, the methods are qualitatively similar, whereas for nonlinear walks, our methods are advantageous.
Non-transformee
E1c
Areo
Cunto
Center
Mternndll dy H0b 12-3
Maerineity #D
Inni 16
steps
grae. n1- 263
Car grae. nc 28.3
finas
Der Noar une. 22.3
0.
(a) Zoom in
(b) Shift
(c) Shift Y
Figure 4: Quantitative comparison with (Jahanian et al. 2020). We show the probability densities of object areas and locations after 2 (top) and 5 (bottom) steps of walks for BigGAN- 128. The step- size is the same for the linear walks, and matches the size of the first step of the nonlinear walk. Our walks have similar effects those of Jahanian al. (2020), with the nonlinear variants achieving lower FID scores after 5 steps, at the cost of only slightly weaker transformation effects.
We assume again that E[z] O, and make the additional assumption that E[xx"] I²I, which is the case in all current GAN frameworks. In this setting, the objective in (5) reduces
d? [WWp(WMMM)) +I (I-
(6)
5

---

Published as a conference paper at ICLR 2021
Our
Our
Creatt
7
Steersiliity
Our Sloerubility Neutron
3
(a) Linear trajectories
(b) Nonlinear trajectories
Figure 5: Endpoints. (a) Linear walks eventually lead to deteriorated images (shown here for zoom). (b) Our nonlinear walks converge to meaningful images. The nonlinear trajectories of the GAN steerability method (Jahanian et al. 2020) also converge, but always to the same (unnatural) image for a given class.
where || ||r denotes the Frobenius norm. Here, I appears only in the second term, which identical the second term of (2). Therefore, the optimal q as (3). The matrix M appears only in the first term, which is easily shown to be minimized when setting the diagonal entries of M to
w D
M.
w D w:
where w; is the ith column W.
Controlling the step size opposed linear trajectories, refining the step size along our curved trajectories necessitates modifying both M and q. To do so, we can search for matrix M and vector with which N steps of the form 2n+1 M zn + q are equivalent walk Noting that the Nth step of the refined walk can be explicitly written NN M zg + 0= Mz )q. we conclude that the parameters of this N-times finer walk are
single step of the
N-1 Σ
M-MM
M
q.
Convergence point the spectral norm of M is smaller than 1, then we have that
n-1
lim 2n lim 7-000 10000
M" Mm 20 +
E M*
Σ
-I-
where used the fact that the first term tends Superficially, this may seem to imply that the endpoint of the trajectory is not a function of the initial point Zg. However, recall that in hierarchical architectures, like BigGAN, z refers the part of the latent vector that enters the first layer. The rest of the latent vector is not modified throughout the walk. Therefore, the latent vector the endpoint equals the latent vector of the initial point, except for its subset of entries corresponding to the first hicrarchy level, which are replaced by (I- MM q.
and the second term is Newmann series.
2.3 ACCOUNTING FOR FIRST-ORDER DATASET BIASES VIA GREAT CIRCLE TRAJECTORIES
In the Neumann walk, the step size decreases along the path (as ||z,+1 2..|l - 0). We now discuss an alternative nonlinear trajectory that has natural endpoint yet permits a constant step size. Here we avoid low density regions by explicitly requiring that the likelihood of all images along the path constant. For (O, I). this translates the requirement that the whole trajectory lie on the sphere whose radius equals the norm of the original latent code Zg. We stress that the method we discuss here can be applied any direction q, whether determined in supervised manner not.
Specifically, suppose we want to steer our latent code towards a normalized direction v = q ///g/]- Then we can walk along the great circle on the sphere that passes through our initial point 2o. and

---

Published as a conference paper at ICLR 2021
the point || v (blue circle in Fig. 6 Mathematically, let V denote the (one-dimensional) subspace spanned by v and let Py = vo" and P = I Py denote the orthogonal projections onto V and V respectively. Then the great circle trajectory can be expressed as
nn | |l (u cos(nA 0) sin(nA 0)),
(10)
where Pyt 20/ /oll Py- zoll and 0 arccos( P 20 2 | I) sign( (20, v)). The effect of this trajectory for a zoom-in direction shown in Fig. 6 (third row). The natural endpoint of the great- circle path is || |//l/e (blue point), beyond which the contribution of starts decrease. As seen in Fig. [ this endpoint indeed corresponds to a plausible zoomed-in version of the original image.
2.4 COMPARISON
Figure ()) compares our nonlinear walks (Neumann and great-circle) with those of the GAN steer- abilty work Jahanian al. (2020). As be seen, the latter tend involve undesired brightness changes. The advantage of our nonlinear trajectories over the linear ones becomes apparent when performing long walks, as exemplified in Fig. [ In such settings, the linear trajectories deteriorate, whereas our nonlinear paths have meaningful endpoints. This can also be seen in Fig. 4 which reports the Frechet Inception distances (FID) achieved by the two approaches. Interestingly, the nonlinear trajectories of the GAN steerability method also have endpoints, but these endpoints are the same for all images of certain class (and distorted).
3 UNSUPERVISED EXPLORATION OF TRANSFORMATIOOS
To go beyond simple user-prescribed geometric transformations, we now discuss exploration of additional manipulations in an unsupervised manner. The key feature of our approach is that by revealing a large set of directions, we can now also account for second-orde dataset biases.
3.1 PRINCIPAL LATENT SPACE DIRECTIONS
We start by seeking a set of orthonormal directions (possibly a different set for each generator hierarchy) that lead to the maximal change at the output of the layer which is injected. These directions are precisely the right singular vectors of the corresponding weight matrix W. i.c., the kth most significant direction is the kth column of the matrix V in the singular value decomposition W = USV (assuming the diagonal entries of S are arranged in decreasing order). This reveals directions corresponding to many geometric, texture, color, and background effects (see Fig.
Our approach is seemingly similar to GANspace (Härkönen et al., 2020)) which computes PCA of activations within the network. However, they op- timize over latent space directions that best map to this deep PCA basis. Con- cretely, they feed-forward random latent codes (2(1)} to obtain deep- featufe rep- resentations {)("),, ), compute the PCA
Method
Memory 0
Time
Jahanian et al. (2020)
40 min (per dir.)
Harkonen et al. (020)
1GB
14 hrs (all)
Voynov & Bahenko
2020
0
10 hrs (all)
Our principal directions
0
327 (all)
Table 1: Complexity for BigGAN-deep-512.
basis A and mean vector μ of these features, and then solve for a steering basis V arg min E, ||V A (g - 4)) - =(]]]. Thus, besides computational inefficiency (see Tab. they obtain set of orthoooollaa latent-space directions (see App. Fig. 48) that correspond re- peated effects (see App. Figs.42 42-4).2.6). In contrast, our directions are orthogonal by construction, and therefore capture a more diverse set of effects (see App. Figs.42-46). For example, the semantic dis- similarity between G(z) and G(2 + 3v) is 64% larger with our method, measured by the average LPIPS distance (Zhang et al. 2018 over the first 50 directions (33 10 -0 for GANSpace, 54 10-3 for us).
Having determined a set of semantic directions, we now want to construct trajectories that exhibit the corresponding effects, but also account for dataset biases. As discussed in Sec. and illustrated in the first two rows Fig. 6 performing linear walks along these directions eventually leads to distorted images. A more appropriate choice is thus to use the great-circle walk described in Sec.

---

Published as a conference paper at ICLR 2021
Nose coord.
GAN- Space
Lincur
Our Great
V
Our Small
Dref
Steps
Figure 6: Orbits in latent space. A linear trajectory (magenta) in the principal direction v cor- responding to zoom, eventually draws apart from the sphere and results in distorted images. The great circle (blue) that connects zu with ||z keeps the image natural all the way, but allows also other transformations (shift in this case). The small circle (green) that only modifies to v, does not induce any other transformation besides zoom (Ver is the least dominant direction). Particularly, it keeps the nose's vertical coordinate fixed (right plots). See also App. Figs. 40-41
in addition
This is illustrated in the third row of Fig. 6 While leading to meaningful endpoints, a limitation of the great circle trajectory is that when walking on the sphere towards e, we actually also modify the projections onto other principal directions. This causes other properties to change besides the desired attribute. For example, in Fig. 6 the great circle causes shift, centering the dog in addition to the principal zoom effect (see the nose position graphs on the right). This stems from a second- order dataset bias. Indeed, as shown in Fig. 7 BigGAN generates small (zoomed-out) dogs at almost any location within the image, but its generated large (zoomed-in) dogs tend to be centered.
3.2 ACCOUNTING FOR SECOND-ORDER DATASET BIASES VIA SMALL CIRCLE TRAJECTORIES
Using our set of directions to battle second-order biases is non-trivial, as walking on the sphere towards while keeping the projections onto all other principal directions fixed is impossible (it induces too many constraints). However, we note that if we allow the projection onto only one of the other directions, say thet, change, then it becomes possible to keep the projections onto all other axes fixed. Such a trajectory is in fact a small circle on the sphere, that lies in the affine subspace that contains 2g and is parallel V span (v, eer) Specifically, the small circle walk is given by
Zn = PyLL 20 + Py (Wier cos(n + 0) + sin(n + O));
(1I)
x sign( (Pyzo, v)) with Pvi. = viero One natural choice for Viet is the principal direction having the smallest singular value, which corresponds to the weakest effect. As can be seen in the bottom row of Fig. 6 the small circle trajectory with this choice leads to zoom effect without shift or any other dominant transformation. This is also illustrated in Fig, [ which shows the distribution of the horizontal translation between the initial point and the endpoint of the trajectory. As can be seen, the small circle walk incurs the smallest shift and keeps the FIID highest, albeit leading to a slightly smaller zoom effect. In App. A.2 we show additional examples, including with differrntt choices of Vref-
where = arccos( Pv,., 20
4 ATTRIBUTE TRANSFER
In the previous sections we explicitly computed directions in latent space. An alternative way of achieving desired effect, is to transfer attributes from a different image. As we now show, this can also be achieved without optimization. Specifically, in App. A.2 we show that for BigGAN, princi- pal directions corresponding to different hierarchies control distinctively different attributes. Now, key observation is that this allows transferring attributes between images, simply by copying from a target image the part of Z corresponding to a particular hierarchy (see Fig, (8. For example, to transfer pose, replace the part corresponding to the first level. As seen in Figs. and g this al-
8

---

Published as a conference paper at ICLR 2021
FID 10.2
FTD =
25.2
FID 26.1
FTD 15.5
FTD - 12.8
0.1
Rei Horizontal Shift
Rei Palattive Horaonal Shift
Rel Horizontal Shift
Rel Pellative Horioonal Shift
02
02
22
83
92
64
Area
Area
Area
Area
Area
Model
(b) Stecrabiiity
(c) GAaCeace
(d) Our circle
(e) small circle
Figure 7: Accounting for second-order dataset bias. In red is the joint distribution of area and horizontal center of BigGANN-geneatte Labrador dogs. This plot shows that zoomed-out dogs can appear anywhere, whereas zoomed-in dogs are mostly centered. In blue are the joint distributions of area and horizontal translation (namely delta shift) achieved by walks in zoom-in direction. All walks indeed increase the area, but also undesirably shift the dog. Our methods incur smaller shifts, with the small circle walk incurring negligible shift. From left the right, the mean shifts of the methods are 0.08, 0010, 0.06 and 0.01. This allows us achieve lower FIDs, but the cost of achieving slightly smaller zoom effects (the mean areas are 0.85, 0.83, 0.80 and 0.76).
Trunsforned seloritexture
Target
Texture
Target
Random Sample
Trunsfered
code
parts (chunks)
Target
Pose
Trans ferred
pose
Trunsfered
Figure 8: Attributes transfer. In BigGAN-128 the latent code is divided into chunks that are injected to different hierarchy levels. Transferring pose, color or texture, can be done by copying specific parts of the latent code from the target image.
lows transferring pose even across classes. Within the same class, we can transfer color by copying the elements of hierarchies 4,5 and and texture by copying hierarchies 3,4 and 5 (see Appendix for more examples). Note that unlike other works discussing semantic style hierarchies (e.g., (Karras) et al. 2019a Yang et al. 2019),, our pre-trained BigGAN was not trained to disentangle attributes.
CONCLUSION
We presented methods for determining paths in the latent spaces of pre-trained GANs, which corre- spond to semantically meaningful transformations. Our approach extracts those trajectories directly from the generator's 's weights, without requiring optimization or training of any sort. Our methods significantly more efficient than existing techniques, they determine larger of distinctive semantic directions, and are the first to allow explicitly accounting for dataset biases.
Acknowledgements This research was supported by the Israel Science Foundation (grant 852/17) and by the Technion Ollendorff Minerva Center.

---

Published as a conference paper at ICLR 2021
REFERENCES
Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene genera- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4561-4569, 2019.
Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam. Optimizing the latent space of generative networks. In Proceedings of the 35th International Conference on Machine Learn- ing. pp. 600-609, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2018.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9789-8797, 2018.
Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru. Detecting bias with genera- tive counterfactual face attribute augmentation. arXiv preprint arXiv: /906. 06406, 2019.
Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defini- tions of cognitive image properties. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. arXiv preprint arXiv:2004.02540, 2020.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. In International Conference on Learning Representations, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Proc. International Conference Learning Repre- sentations (ICLR), 2018
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2019a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. arXiv preprint 19Xiv:1912.0495 2019b.
Yannic Kilcher, Aurelien Lucchi, and Thomas Hofmann. Semantic interpolation in implicit models. In International Conference on Learning Representations, 2018
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representatiins, 2018.
William Peebles, John Peebles, Jun- Yan Zhu, Alexei A. Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. In International Conference on Learning Representations, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1511::5511.064 1.06434, 2015.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020.
10

---

Published as a conference paper at ICLR 2021
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. arXiv preprint arXiv:2002.03754, 2020.
Tom White. Sampling generative networks. arXiv preprint MxXiv:66090044000 2016.
Taihong Xiao, Jiapeng Hong, and Jinwen Ma. Elegant: Exchanging latent encodings with gan for transferring multiple face attributes. Proceedings of the European conference on computer vision (ECCV), pp. 168-184 2018.
Ceyuan Yang, Yujun Shen, and Bolei Zhou. Semantic hierarchy emerges in deep generative repre- sentations for scene synthesis. arXiv preprint arXiv:1911.09267, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.

---

Published as a conference paper at ICLR 2021
A GAN "STEERABILITY" WITHOUT OPTIMIZATION: APPENDIX
A. QUANTITATIVE EVALUATION
We adopt the method proposed in Jahanian et al. (020) and utilize the MobileNet-SSD-VV detectof to estimate object bounding boxes. To quantify shifts, we extract the centers of the bounding boxes along the corresponding axis. To quantify zoom, we use the area of the bonding boxes. In the following paragraphs, we elaborate on all quantitative evaluations reported in the main text.
Figure Here, we show the probability densities of object areas and locations after (top) and 5 (bottom) steps. Since we use unit-norm direction vectors, the length of the linear paths we walk through are 2 and 5 as well. As for the nonlinear path, we choose the first step to have the same length. However, the overall length of the path is different. For example, the nonlinear trajectory have a total length of 5.95, but reach at point of distance only 4.3 from the initial point. We used 100 randomly chosen classes from the ImageNet dataset, and 30k images from each class. The same images are used for both the FID measurement and for generating the PDF plots.
average, five steps of
Figure 6 In order to ensure that each step of the linear walk and the great and small circle walks has the same geodesic distance, we set
A = Aa||zoll As Pvzoll.
(12)
where AL. AG and As are the step sizes of the linear, great circle and small circle walks, respec- tively. This ensures that the arc-length of a step on the circles is the same as the length of a step of the linear walk.
Figure 7 Here, we aim to demonstrate a particular second order dataset bias. We chose 10 classes which we found to exhibit strong coupling between the size and location of the object. For example, dogs, cats and in general, animals. We plotted 80 levels-sets of 2D KDEs computed using the seaborne package. In Fig. 7 we show results for a Labrador retriever dog, we observed similar results for the classes: golden retriever (207), Welsh springer spaniel (218), Great grey Owl (24), Persian cat (283), plane (726), tiger (292), Old English sheepdog (229), passenger car (705), goose (99), husky (248). See Figs. 40 and 41 for additional results.
Table In Tab. 1 we compare the running time and memory usage of all methods. For JJhanian al. 2020). we measure the time it takes to learn one direction, which includes the training process. For Harrkönen et al. 2020). we measure the total time it takes to extract the directions, including the sample collection, the PCA, and the regression. We noticed that the regression stage was the heaviest. As for our method, we measure the time it takes the CPU to perform SVD. The column "Memory" specifies the required memory for collecting samples. Only GANSpace (Härkönen al.) 2020) requires that stage.
A.2 UNSUPERVISED EXPLORATION OF PRINCIPAL DIRECTIONS
A.2.1 COMPARISONS WITH RANDOM DIRECTIONS
In Fig. 10 9 we explore principal directions via linear walks, using the same initial image (in the middle). Figs. 14 16 we explore the transformations that arise in each hierarchy of BigGAN-129 Specifically, we compare our linear directions which are based on SVD, with random directions. We draw different directions from an isotropic Gaussian distribution and normalize them to have unit-norms, similarly to our directions. Then, we linearly add them to the initial latent code with fixed number of steps. We can observe that each random direction induces a different complex effect, which cannot be described by single semantic property. For examples, in the first random direction (RI) we can see rotation, zoom and background changes, while in the third (R3), there is a kind of vertical shift. On the other hand, our principal directions show one prominent transformation
12

---

Published as a conference paper at ICLR 2021
for each scale. We focus directions that have the same effect for all classes and do not show directions that lead to different effects for different classes, like changes of day-night in one class and background in another class.
A.2.2 ALTERNATIVE SMALL CIRCLE WALKS
show more examples, this time with small circle walks towards principal direc- tions. all those examples, the reference direction Uht for the small circle, is the least dominant direction (namely, the singular vector with the smallest singular value). This ensures that when walk- ing towards the principal direction v, we modify no other dominant property. That is, we modify the property associated with without modifying the properties associated with any other principal direction, besides Vier (which is the least dominant one). In Figs. 33 we show some cases in which the initial generated image is in the middle of the small circle path and therefore in these cases, we need to take different number of steps to each side. The endpoints are defined as the points where the cosine in Eq. becomes 0 and We do not have to choose the reference direction Uref to be the least dominant one. If we choose it to be a dominant direction, then we may obtain various interesting phenomena, depending on the in- teraction between the directions and Vred- This is illustrated in Figs. 37 Specifically, in Fig. 37 and 38 we perform a walk in the direction corresponding to zoom, while allowing only the vertical achieve significant
In Figs. 29 33
shift change. In this case, the walk manages center the object zoom effect. In Fig. (9) on the other hand, we perform a walk the direction corresponding to zoom while allowing only the rotation change. Here, the zoom effect less dominant, but do see a strong rotation effect.
A.2.3 SECOND ORDER DATASET BIASES
In Figs. 40 and 41 we show more examples for second order dataset biases. Specifically, those figures depict the joint distributions of and horizontal center shift (top) and area and vertical center shift (bottom) at the end of walks that are supposed to induce only zoom-in. Our small circle walks exhibit the smallest undesired shifts.
A.2.4 COMPARISONS WITH GANSPACE
show visual comparisons with GANSpace (Härkönen et al. 2020). We specifically focus on the first 50 directions founded by each method and show that our linear directions lead to stronger effects for most of the directions. All directions were scaled have unit norm and linearly added or subtracted from the initial latent code with the same step size. In Fig. [48] we show that our direction are orthogonal to each other much more then the directions found by (Härkönen
In Fig. 42 46
et al. 2020
A.3 USER PRESCRIBED SPATIAL MANIPULATIONS
We provide additional examples for walks corresponding to user prescribed geometric transforma- tions. We focus on zoom, vertical shift and horizontal shift, and show both linear and our nonlinear trajectories.
A.3.1 COMPARISONS WITH JAHANIAN ET AL.
In Figs. we show additional comparisons with Jahanian et al. (2020)
A.3.2
ADDITIONAL RESULTS
we show additional zoom trajectories and and in Fig. 54 55 tories. As be seen, the linear trajectories often remain more loyal to the original image (at the center) after a small number of steps. However, for large number of steps, the nonlinear trajectories lead to plausible images.
In Figs. 50 53
additional shift trajec-
13

---

Published as a conference paper at ICLR 2021
A.3.3 RESULTS ON DCGAN
In Figs. 56 we show results with ResNet based GAN presented in Miyato al. (2018). That GAN has a FC layer as the first stage, which is all we need in order to perform our spatial manip- ulations and extract principal components. Since that architecture is not hierarchical one, we can manipulate the first layer only.
14

---

Published as a conference paper at ICLR 2021
Zoom
Shift
Rotate
1 hierarchy
Zoom
2nd hierarchy
Background
3rd hierarchy
Background Removal
Lighting ("Shadowing")
4th hierarchy
Brightness
5th hierarchy
Green
Temperature
Yellow
Red
6th hierarchy
Figure 9: Our explored directions in BigGAN
15

---

Published as a conference paper at ICLR 2021
Zoom
Shift up
Rotate
1 hierarchy
Zoom
2nd hierarchy
Background clutter
3rd hierarchy
Background Removal
Lighting ("Shadowing")
BW
4th hierarchy
Brightness
5th hierarchy
Green
Temperature
Yellow
Red
6th hierarchy
Figure 10: Our explored directions in BigGAN.
16

---

Published as a conference paper at ICLR 2021
R2
R3
R4
R5
Zoom
Rotate
Shift Y
Scale
RI
R2
R3
R4
Zoom
2nd Scale
Our vs. random directions. We illustrate the effects of five random directions RI,.... R5 (normally distributed and scaled have unit norms) in the first and second scales of BigGAN, in comparison with our principal directions. We can see that each random direction leads different changes, but is impossible associate single dominant property with each direction. For example, in R5 we can see changes in size, location, and pose. This is while our directions separate those effects into unique paths.
Figure 11:
17

---

Published as a conference paper at ICLR 2021
RI
R3
R5
Background Clutter
3rd Scale
RI
R2
R3
R4
R5
Background Removal
Lighting ("Shadowing")
BW
4th Scale
Figure 12: Our vs. random directions. We show the effects of five random directions in the third and fourth scales of BigGAN in comparison with our principal directions.
18

---

Published as a conference paper at ICLR 2021
RI
R2
R3
R4
R5
Brightness
5th Scale
R2
R5
Temperature
Yellow
Red
6th Scale
Figure 13: Our vs. random directions. We show the effects of five random directions in the fifth and sixth scales of BigGAN in comparison with our principal directions.
19

---

Published as a conference paper at ICLR 2021
RI
R2
R3
R4
R5
Zoom
Rotate
Shift Y
1 Scale
RI
R2
R3
R4
Zoom
2nd Scale
Figure 14: Our vs. random directions. We illustrate the effects of five random directions RI,.... R5 (normally distributed and scaled have unit norms) in the first and second scales of BigGAN, in comparison with our principal directions. We can see that each random direction leads different changes, but is impossible associate single dominant property with each direction. For example, in R5 we can see changes in size, location, and pose. This is while our directions separate those effects into unique paths.
20

---

Published as a conference paper at ICLR 2021
RI
R3
R4
R5
Background Clutter
3rd Scale
RI
R2
R3
R4
R5
Background Removal
Lighting ("Shadowing")
BW
4 Scale
Figure 15: Our vs. random directions. We show the effects of five random directions in the third and fourth scales of BigGAN in comparison with our principal directions.
21

---

Published as a conference paper at ICLR 2021
RI
R2
R3
R4
R5
Brightness
5th Scale
R2
R5
Green
Temperature
Yellow
Red
6th Scale
Figure 16: Our vs. random directions. We show the effects of five random directions in the fifth and sixth scales of BigGAN in comparison to our principal directions.
22

---

Published as a conference paper at ICLR 2021
R2
R3
R4
R5
Zoom
Rotate
Shift Y
Scale
RI
R2
R3
R4
Zoom
2nd Scale
Our vs. random directions. We illustrate the effects of five random directions RI,... R5 (normally distributed and scaled have unit norms) in the first and second scales of BigGAN, in comparison with our principal directions. We can see that each random direction leads different changes, but is impossible associate single dominant property with each direction. For example, in R5 we can see changes in size, location, and pose. This is while our directions separate those effects into unique paths.
Figure 17:
23

---

Published as a conference paper at ICLR 2021
RI
R3
R5
Background Clutter
3rd Scale
RI
R2
R3
R4
R5
Background Removal
Lighting ("Shadowing")
BW
4 Scale
Figure 18: Our vs. random directions. We show the effects of five random directions in the third and fourth scales of BigGAN in comparison with our principal directions.
24

---

Published as a conference paper at ICLR 2021
RI
R2
R3
R4
R5
Brightness
5th Scale
R2
R5
Temperature
Yellow
Red
6th Scale
Figure 19: Our vs. random directions. We show the effects of five random directions in the fifth and sixth scales of BigGAN in comparison our principal directions.
25

---

Published as a conference paper at ICLR 2021
Scale Direction Zoom
Figure 20: 1st principal direction of the first scale BigGAN
26

---

Published as a conference paper at ICLR 2021
Scale Direction Shift up
6
Figure 21: 2nd principal direction of the first scale in BigGAN
27

---

Published as a conference paper at ICLR 2021
Scale Direction 3D rotate
Figure 22: 3rd principal direction of the first scale in BigGAN
28

---

Published as a conference paper at ICLR 2021
Scale 3. Direction I Background manipulation Taackgroo Dotterr
Figure 23: 1st principal direction of the third scale in BigGAN
29

---

Published as a conference paper at ICLR 2021
Scale Direction Background Removal
Figure 24: First principal direction of the fourth scale in BigGAN
30

---

Published as a conference paper at ICLR 2021
4, Direction
BW
Figure 25: Second principal direction of the fourth scale in BigGAN
31

---

Published as a conference paper at ICLR 2021
Scale 4, Direction Lighting ("Shadowing"
Figure 26: Third principal direction of the fourth scale in BigGAN
32

---

Published as a conference paper at ICLR 2021
Scale 5, Direction Brightness
Figure 27: First principal direction of the fifth scale in BigGAN
Yellow
Temperature
Scate 6
Green
Red
Figure 28: First three principal direction of the sixth scale in BigGAN
33

---

Published as a conference paper at ICLR 2021
Scale 1, Direction 1
Zoom
Figure 29: First principal direction of scale in BigGAN (small cirele walks).
Scale 1, Direction Shift Y
Figure 30: Second principal direction of scale in BigGAN (small circle walks).
Scale 1, Direction 3 Rotate
Figure 31: Third principal direction of scale
BigGAN (small circle walks).
34

---

Published as a conference paper at ICLR 2021
Scale 3. Direction. 3
Color (B&
Figure 32: Third principal direction of scale 4 BigGAN (small circle walks).
Scale Direction Background manipulations
Figure 33: First principal direction of scale (small circle walks). When walking enough steps in the linear direction, total background removal is observed (see Fig[15] However, it might come with slight change of object colors. Therefore, we will not constantly see it within the small circle framework (see last image that bulk in comparison to the other 3).
Brightness (Shatow) (ooo)
Scale 3, Direction 2
Figure 34: Second principal direction of scale 4 in BigGAN (small circle walks).
35

---

Published as a conference paper at ICLR 2021
Scale Direction Vertical Shift
Scale 1, Dirvction 3
3D Rotate
Figure 35: Chosen principal direction of scale 3 in BigGAN (small circle walks). When the initial generated image is not at the middle of the path, we need to take different number of steps to each side.
36

---

Published as a conference paper at ICLR 2021
Scale 3, Direction 3
Black and White
Scale Direction Brightnes ("shadow")
Figure 36: Chosen principal direction of scale I in BigGAN (small circle walks), When the initial generated image is not at the middle of the path, we need to take different number of steps at each side.
Scale 1, Direction (zo0m) at expense of direction 2 (shift Y)
Figure 37: Modifying the second principal direction of scale on the expense of the first principal direction of that scale in BigGAN (small circle walks).
37

---

Published as a conference paper at ICLR 2021
Scale 1, Direction 1 (200m) at
expense of didrrccioon 2 (shift Y)
Figure 38: Modifying the second principal direction of scale on the expense of the first principal direction of that scale in BigGAN (small circle walks).
expense of direction 3 (rotate)
Scale 1, Direction I (zoom)
Figure 39: Modifying the third principal direction of scale on the expense of the first principal direction of that scale in BigGAN (small circle walks).
38

---

Published as a conference paper at ICLR 2021
Hardontal Shift
Horizontal Shift
Horizontal Shift
Horizontal shift
Area
Area
Area
Area
Vertical Shift
Vertical Shift
Vertical Shift
Vertical Shift
-14
Area
Area
Area
Area
Johanian al.
GANSpace
Our great circle
small circle
Figure 40: Second order dataset biases. We explore the coupling between zoom and horizontal translation (top) and zoom and vertical translation (bottom) for Persian cat class BigGAN-deep. can be clearly observed that the small circle path exhibits the smallest undesired shifts when increasing the area.
39

---

Published as a conference paper at ICLR 2021
Hordontal Shift
Horizontal Shift
Horizental Shift
Horizonaal Shift
-12
-22
-04
Area
Area
Area
Area
Vertical Shift
Vertical Shift
Vertical Shift
Vertical shift
2
Area
Area GANSpace
Area
Area
Johanian al.
Our great circle
circle
Figure 41: Second order dataset biases. We explore the coupling between zoom and horizontal translation (top) and zoom and vertical translation (bottom) for husky dogs class BigGAN-deep. can be clearly observed that the small circle path exhibits the smallest undesired shifts when increasing the area.
40

---

Published as a conference paper at ICLR 2021
Random
GANSpace
Our
Figure 42: Comparison with GANSpace and random directions in BigGAN-deep (principal vectors 0-25). The image at the center of each block is the original image. We linearly added the vectors with equal steps. Both directions are normalized to have unit-norms. We can see that our trajectories induce stronger change than those of Härkönen et al. (2020). The averaged LPIPS variance is 0.036 and 0.059 for Härkönen et al. (2020) and our method, respectively.
41

---

Published as a conference paper at ICLR 2021
Random
GANSpace
Our
Figure 43: Comparison with GANSpace and random directions in BigGAN-deep (principal vectors 25-50). The image at the center of each block is the original image. We linearly added the vectors with equal steps. Both directions are normalized have unit-norms. can be observed that our trajectories induce stronger change than those of Härkönen et al. (2020). The averaged LPIPS variance is 0.03 and 0.049 for Härkönen et al. (2020) and our method, respectively.
42

---

Published as a conference paper at ICLR 2021
GANSpace
Our
Figure 44: Comparison with GANSpace in BigGAN-deep (principal directions 0-25). The image at the center of each block is the original image. We linearly added the vectors with equal steps. Both directions are normalized to have unit-norms. can be observed that our trajectories induce stronger change than those of Härkkönn et al. (2020).
43

---

Published as a conference paper at ICLR 2021
GANSpace
Our
Figure 45: Comparison with GANSpace in BigGAN deep (principal directions 25-50).
44

---

Published as a conference paper at ICLR 2021
GANSpace
Our
Figure 46: Comparison with GANSpace in bigGAN deep
(principal directions 25-50).
45

---

Published as a conference paper at ICLR 2021
"Show horizon +
GANSpace
Our, Linear
Our, Small
Figure 47: Comparison with Härkönen et al. (2020). An example for the "show horizon" direction which apply edit only layers 1-5 (Härkönen et al.) 2020) in BigGAN-deep 512. We can that our linear directions achieve similar effects to those of GANSpace (blurring the background).
However, in both cases, we can also slight changes in the object size and pose. On the other hand, when using our small circle walk, we keep the same size and pose.
30623
30003
218/0
30000
3003
32005
34600
30000
32606
wwweagaaii 2.0 5:8 80 Wut.t 888
30000
#wanns9#830 6 600 R03E:DSERCE
a GANSpace
(b) Our
Figure 48: Evaluating orthogonality. We show absolute value of the correlation between every two directions among the first 80 directions in GANspace and our method for BigGAN-deep-512.
46

---

Published as a conference paper at ICLR 2021
Zoom
Zoom
Our
[10]
Our
[10]
[10]
Shift Y
Shift Y
Our
[10]
Our
110]
Our
[10]
Shif
Shift
Our
100
Our
[10]
Our
110]
Linear Trajectorics
Nonlincar Trajectorics
Figure 49; User prescribed transformations with BigGAN.
47

---

Published as a conference paper at ICLR 2021
Linear
Non- linear
Linear
Non- linear
Linear
Non- linear
Figure 50: User prescribed zoom with BigGAN. Our method, linear vs. non-linear trajectories.
48

---

Published as a conference paper at ICLR 2021
Linnea
Non- linear
Linear
Non- linear
Linear
Non- linear
Linear
Non- linear
Linear
Non- linear
Lincar
Non- lincar
Figure 51: User prescribed zoom with BigGAN. Our method, linear vs non-linear trajectories
49

---

Published as a conference paper at ICLR 2021
Linear
Non- linear
Linear
Non- linear
Linear
Non- linear
Linear
Non- linear
Linear
Non- linear
Linear
Non- lincar
Figure 52: User prescribed zoom with BigGAN. Our method, linear vs. non-lin non-linear trajectories.
50

---

Published as a conference paper at ICLR 2021
Linear
Non- lincar
Linear
Non- linear
Linear
Non- linear
Linear
TTERSS
Non- linear
Linear
Non- linear
Linear
Non- lincar
Figure 53: User prescribed zoom with BigGAN. Our method, linear non-linear trajectories.
51

---

Published as a conference paper at ICLR 2021
Linear
Non- lincar
Linear
Non- linear
Linear
Non- lincar
Linear
Non- linear
Linear
Non- linear
Linear
Non- linear
Figure 54: User prescribed vertical shift with BigGAN. Our method, linear vs. non-linear trajecto-
ries.
52

---

Published as a conference paper at ICLR 2021
Linear
linear
Linear
Non- linear
Linear
Non- lincar
Figure 55: User prescribed vertical shift with BigGAN. Our method, linear vs. non-linear trajecto-
rics.
53

---

Published as a conference paper at ICLR 2021
Zoom
-Zoom +
Figure 56: Zoom transformation with DCGAN.
54

---

Published as a conference paper at ICLR 2021
Shift X +
Shift X +
Figure 57: Shift X transformation with DCGAN.
55

---

Published as a conference paper at ICLR 2021
A.4 ATTRIBUTE TRANSFER
We next provide more attribute transfer examples. Figures 60 show pose transfer examples, which are obtained by swapping the part of the latent vector corresponding to scale 1. Figure 61 depict texture transfer examples, which correspond to swapping the parts of the latent vector and the class, corresponding to scales 3,4,5.
56

---

Published as a conference paper at ICLR 2021
Shift Y +
Shift Y +
Figure 58: Shift Y transformation with DCGAN.
57

---

Published as a conference paper at ICLR 2021
Original Images
Target Pose
Transferred
Original Images
Target Pose
Transferred
Figure 59: Pose transfer by swapping scale of the latent vector.
58

---

Published as a conference paper at ICLR 2021
Original Images
Target Pose
Transferred
Figure 60: Pose transfer by swapping scale 1 of the latent vector.
Original Images
Target Texture
Transferred
Figure 61: Texture transfer by swapping scales 3,4,5 of the latent vector.
59


Reviews:
[{"review_id": "us8ffD545_C", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose two new techniques that extract interpretable directions from latent spaces of pretrained GAN generators. Both techniques are very efficient and are shown to work with the state-of-the-art BigGAN models. Furthermore, the authors describe additional details of the method, like determining the transformation end-points, which are important for usage in the practical visual editing.\n\nStrengths:\n\n1. The paper tackles the important problem, provides a thorough description of the field, demonstrates an in-depth understanding of the area. The important small details (determining the end-points, entangled transformations) are only slightly addressed in the existing literature, but are crucial for editing applications. This paper convincingly addresses this gap.\n\n2. The proposed techniques are both simple and efficient, much faster compared to existing methods.\n\nWeaknesses:\n\n1. What upset me most was a certain amount of overclaiming about the main contributions.\n\n\"Second, it detects many more semantic directions than other methods.\" - I cannot agree with this statement.\n\na) For user-specified transformations, the method finds the same directions as  Jahanian et al. (2020).\n\nb) For unsupervisedly discovered transformations, I did not find in the text any examples of directions, that are not covered by the:\n\nVoynov & Babenko, 2020\n\nHa ̈rko ̈nen et al. (2020), 2020,\n\nThe Hessian Penalty: A Weak Prior for Unsupervised Disentanglement, ECCV 2020 (which is a missing related work, btw)\n\nIn my opinion, the statement is misleading, which is not acceptable when listing contributions.\n\n2. I am not convinced by the significance of the search of user-specified geometric transformations, which take the largest part of the submission. Simple transformations, like zoom/shift/brightness, can be easily obtained automatically in editing applications, do we really need GANs for them?\n\n3. \"[Ha ̈rko ̈nen et al. (2020)] obtain a set of non-orthogonal latent-space directions that correspond to repeated effects. In contrast, our directions are orthogonal by construction, and therefore capture a super-set of the effects found by Ha ̈rko ̈nen et al. (2020)\"\n\nThis sound like a bold claim. First, directions from Ha ̈rko ̈nen et al. (2020), while not being orthogonal by construction, de-facto still can be close to orthogonal. Second, from both the main text and appendix, I did not understand, why the authors' method captures a super-set of directions, no quantitative comparison is provided.\n\nTo sum up, my current evaluation is (5). The proposed method is obviously more efficient, but I do not consider this advantage as a very important one, since transformation search is performed only once. Given the weaknesses listed above, I cannot recommend acceptance.\n\nAFTER REBUTTAL:\n\nThe authors have toned down some of their claims and I am increasing my score accordingly. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "tDKpkes4DyO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper studies transformations in GAN latent space that map to meaningful transformations in the generated data. The main contribution is to derive closed form methods for discovering latent transformations that correspond to 1) geometric changes and 2) changes that capture principle components of model variation. The paper also contributes new methods for nonlinear latent transformations, disentangled transformations, and an application to attribute transfer.\n\nI really like this paper. The capabilities demonstrated here aren’t dramatically new — other methods can achieve similar effects — but this paper achieves these effects in a new way, which has its own advantages.\n\nThe positives I see are:\n+ Simple and elegant alternative to prior work on finding latent transformations\n+ Nice qualitative and quantitative results\n+ Practical benefits including speed up, analytical transformation end points, and better disentanglement\n\nMy main criticisms are:\n- toward the end of the paper it’s a bit of a hodgepodge of ideas\n- the ideas and methods in Section 3 are mostly disjoint from those in Section 2\n- the attribute transfer application especially feels tangential, and receives minimal analysis or evaluation\n- the experiment in Figure 7 is not fully convincing\n\nI think everything in this paper is interesting, but the different sections don’t fully cohere together. Section 2 is great and seems to tell a complete story, about closed form solutions to finding geometric transformations. Section 3 then diverges into a few different directions, which don’t directly build on each other. When the circular trajectories are introduced, I’m left wondering: why not use these in Section 2? For example, would it be possible to derive closed form solutions for great circles that map to target transformations $\\mathbf{P}$?\n\nI do like the story about first and second order biases, which somewhat connects Sections 2 and 3. However I would like to see more connections made at the methodological level, or a clear justification for why methods from Section 2 where not used in Section 3 and vice versa.\n\nAside from the coherence of the story, there are a few smaller things that could be improved:\n\nIn Figure 4, it would be nice to show the data distribution, or the distribution from the non-transformed GAN samples. Otherwise it’s hard to tell if the transformations had an effect.\n\nIn Section 3.1 I think an equation would help clarify things, e.g., write out the SVD and refer to the right singular vectors by an algebraic symbol. Perhaps also contrast this with the equation for PCA from Harkonen et al.\n\nThe experiment in Figure 7 is not convincing to me. From the plots it’s hard to tell if the proposed methods are actually incurring less shift. Maybe plotting the delta from the original distribution would make the plot clearer? Most of all, I think some numerical metric should be defined to quantify the second-order biases, rather than just requiring visual inspection of the plots. It is convincing that the FID improves for the proposed methods, but not that the level of disentanglement between zoom and shift improves.\n\nMinor comments:\n1. Page 1: “the precise same effect” — I think this is an overstatement. There are substantial differences between how a single direction affects different classes. For example see Fig 4 of Jahanian et al. 2020. I agree that it is interesting that the directions have _similar_ effects across classes.\n2. In Eqn 3, what if the matrix is singular? Do you use the pseudoinverse?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "saDR2DEadvb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is clearly written. Its significance lies in transferring both user-prescribed transformations and unsupervisingly discovered transformations over a pre-trained generator’s weights to the generator output.  Also, it discovers endpoints for latent space trajectories.\n\nPros: \nOne nice feature of the  proposed technique is that it works on a pre-trained generator, and no extra training or optimization is required, therefore, it is efficient.\nEndpoints of the walks are investigated through analyzing the convergence of the walks.\nFurther demonstrations with walks on the great circle and small circle on the sphere through an initial point  and a  principal direction are presented that give insights into the transformations obtained in the generated images.\n\nCons: \n- Results in the paper display artifacts (e.g Figure 3, Figure 6), therefore raises the question of carefully picked results being displayed, indeed it’s hard to predict the overall quality of generations.\n- Section 3.4 where the attribute transfer is described is not clearly written. Reading this paper, one is not able to understand how background or texture  transfer is obtained through this work. I believe that part is based on a recent work by (Voynov, Babenko 2020). They showed a technique to discover meaningful latent directions in an unsupervised way, which included similar attribute transfers this paper has shown. I believe this paper relies on that work in finding the  directions that correspond to semantically meaningful attributes, or this point was not very clearly explained.  If that’s the case, the contribution of this paper mainly boils down to certain geometric transforms or principal directions.\n\nThe method of (Voynov and Babenko 2020) also operated on a pre-trained generator and obtained semantically meaningful directions in an unsupervised way through only optimizing two other simple networks to obtain meaningful directions. As far as I understand their optimization is not very demanding at all, therefore, is also efficient. Hence, the point made here in terms of ultimate efficiency with no training leads to a trade-off in not being able to discover a richer set of transformations.", "extra_scores": {}, "extra_text": {}}, {"review_id": "oSmWSCE5KN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am a picky reviewer in most cases. This submission is one of the highest-quality manuscripts I have reviewed. The problem setup is clear, the survey is comprehensive, and the contrastive to the prior arts is clear. I am pretty enjoyed reading this paper. \n\nThe task of finding meaningful steerability within the latent space is relatively a new research area. The authors have made a fairly complete review and investigation on the related works, then propose an unsupervised (meanwhile, still providing slight user controllability with a small set of transformations) and training-free algorithm with high-diversity and high-quality (in terms of the robustness of the trajectory found by the algorithm). Besides, the authors propose the concept of the first-order and second-order dataset biases, and tackles the problem with non-linear trajectories discovery. Also, highlight that the proposed method is significantly faster than the prior arts as shown in Table 1. \n\nExperiment-wise, the authors provide a huge set of qualitative results in the appendix. Though the results are already convincing enough, I would still recommend the authors to provide some quantitative numbers for the comparisons on the diversity of latent directions against other methods (Figure 34-38). The visual differences are obvious, but a quantitative number can further showcase the universality of higher diversity. Some simple measurements like a mean over the variance of LPIPS among all dimensions [1] may be fair enough.\n\nDespite the submission is pretty strong and I pretty much has no problem with it, I ended up rating it with an 8, as the impact is relatively narrow in a specific area/task. But I am willing to see the feedback from other reviewers and give it a certain level of adjustment.\n\n\n[1] To avoid ambiguity, I mean \n```\nall_vars = []\nfor trajectory in all_trajectories:\n\timg_pairs = # interpolate the trajectory\n\tcur_var = np.var([lpips(*img_pair) for img_pairs in latent_pairs])\n\tall_vars.append(cur_var)\nmean_var = np.mean(all_vars)\n```\nThis is just a demonstration, any implementation with a similar concept is fine.\n\n\n**[Minor comments]**\n1. While mentioning other related methods in the paper (e.g., Table 1), the references are made with plain text (without hyperlinks) and mostly using the authors' names in replacement of the method names. It is quite hard to keep tracking which reference corresponds to which method. It would be great if the authors can investigate this problem a bit.\n\n2. Typo: Frobenious => Frobenius\n\n**[Discussion]**\nI do not count this section as a part of the review, so it wouldn't affect my scoring if the authors do not reply. \nWhat is the main application of the dataset summarization technique (let's exclude image editing and content creation, which can be achieved with other techniques)? It is interesting to visualize these properties of the dataset, however, the real-world application of such a technique is not obvious and carefully investigated. \nA quick answer might be finding the dataset bias or domain shifting, and we may explicitly fix it when these problems are revealed by these dataset summarization techniques. However, the works on dataset summarization have never explicitly shown if these applications are viable with some experimental setup. \nI am interested in what is the authors' perspective on the applications, and whether showing these applications is an important part of the dataset summarization research direction, as the applications often influence the impact and research direction of proposed new settings and new components.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper gives an elegant and efficient closed form solution for steering directions in the latent space of a pretrained GAN to to produce transformations in the image domain such as scaling and rotation etc,  this also extended to attribute transfer. The new method leads to "speed up, analytical transformation end points, and better disentanglement"  w.r.t to competitive methods.  All reviewers agreed on the merits of this work, and the good qualitative and quantitative results . The rebuttal addressed reviewers questions and concerns regarding the structure of the paper and its coherence.  Accept
Decision: Accept (Spotlight)