GAN "Steerability" without optimization 

1 INTRODUCTION . Since their introduction by Goodfellow et al . ( 2014 ) , generative adversarial networks ( GANs ) have seen remarkable progress , with current models capable of generating samples of very high quality ( Brock et al. , 2018 ; Karras et al. , 2019a ; 2018 ; 2019b ) . In recent years , particular effort has been invested in constructing controllable models , which allow manipulating attributes of the generated images . These range from disentangled models for controlling e.g. , the hair color or gender of facial images ( Karras et al. , 2019a ; b ; Choi et al. , 2018 ) , to models that even allow specifying object relations ( Ashual & Wolf , 2019 ) . Most recently , it has been demonstrated that GANs trained without explicitly enforcing disentanglement , can also be easily “ steered ” ( Jahanian et al. , 2020 ; Plumerault et al. , 2020 ) . These methods can determine semantically meaningful linear directions in the latent space of a pre-trained GAN , which correspond to various different image transformations , such as zoom , horizontal/vertical shift , in-plane rotation , brightness , redness , blueness , etc . Interestingly , a walk in the revealed directions typically has a similar effect across all object categories that the GAN can generate , from animals to man-made objects . To detect such latent-space directions , the methods of Jahanian et al . ( 2020 ) and Plumerault et al . ( 2020 ) require a training procedure that limits them to transformations for which synthetic images can be produced for supervision ( e.g. , shift or zoom ) . Other works have recently presented unsupervised techniques for exposing meaningful directions ( Voynov & Babenko , 2020 ; Härkönen et al. , 2020 ; Peebles et al. , 2020 ) . These methods can go beyond simple user-specified transformations , but also require optimization or training of some sort ( e.g. , drawing random samples in latent space ) . In this paper , we show that for most popular generator architectures , it is possible to determine meaningful latent space trajectories directly from the generator ’ s weights without performing any kind of training or optimization . As illustrated in Fig . 1 , our approach supports both simple userdefined geometric transformations , such as shift and zoom , and unsupervised exploration of directions that typically reveals more complex controls , like the 3D pose of the camera or the blur of the background . We also discuss how to achieve attribute transfer between images , even across object categories ( see Fig . 1 ) , again without any training . We illustrate results mainly on BigGAN , which is class-conditional , but our trajectories are class-agnostic . Our approach is advantageous over existing methods in several respects . First , it is 104×-105× faster . Second , it seems to detect more semantic directions than other methods . And third , it allows explicitly accounting for dataset biases . First order dataset biases As pointed out by Jahanian et al . ( 2020 ) , dataset biases affect the extent to which a pre-trained generator can accommodate different transformations . For example , if all objects in the training set are centered , then no walk in latent space typically allows shifting an object too much without incurring degradation . This implies that a “ steering ” latent-space trajectory should have an end-point . Our nonlinear trajectories indeed possess such convergence points , which correspond to the maximally-transformed versions of the images at the beginning of the trajectories . Conveniently , the end-point can be computed in closed form , so that we can directly jump to the maximally-transformed image without performing a gradual walk . Second order dataset biases Dataset biases can also lead to coupling between transformations . For example , in many datasets zoomed-out objects can appear anywhere within the image , while zoomed-in objects are always centered . In this case , trying to apply a zoom transformation may also result in an undesired shift so as to center the enlarged object . Our unsupervised method allows controlling the extent to which transformation A comes on the expense of transformation B . 1.1 RELATED WORK . Walks in latent space Many works use walks in a GAN ’ s latent space to achieve various effects ( e.g. , ( Shen et al. , 2020 ; Radford et al. , 2015 ; Karras et al. , 2018 ; 2019b ; Denton et al. , 2019 ; Xiao et al. , 2018 ; Goetschalckx et al. , 2019 ) ) . The recent works of Jahanian et al . ( 2020 ) and Plumerault et al . ( 2020 ) specifically focus on determining trajectories which lead to simple user-specified transformations , by employing optimization through the ( pre-trained ) generator . Voynov & Babenko ( 2020 ) proposed an unsupervised approach for revealing dominant directions in latent space . This technique reveals more complex transformations , such as background blur and background removal , yet it also relies on optimization . Most recently , the work of Härkönen et al . ( 2020 ) studied unsupervised discovery of meaningful directions by using PCA on deep features of the generator . The method seeks linear directions in latent space that best map to those deep PCA vectors , and results in a set of non-orthogonal directions . Similarly to the other methods , it also requires a very demanding training procedure ( drawing random latent codes and regressing the latent directions ) , which can take a day for models like BigGAN . Nonlinear walks in latent space Linear latent-space trajectories may arrive at regions where the probability density is low . To avoid this , some methods proposed to replace the popular Gaussian latent space distribution by other priors ( Kilcher et al. , 2018 ) , or to optimize the generator together with the latent space ( Bojanowski et al. , 2018 ) . Others suggested to use nonlinear walks in latent space that avoid low-probability regions . For example , Jahanian et al . ( 2020 ) explored nonlinear trajectories parametrized by two-layer neural networks , while White ( 2016 ) proposed spherical paths for interpolating between two latent codes . Hierarchical GAN architectures Recently there is tendency towards hierarchical GAN architectures ( Karras et al. , 2018 ; 2019a ; Brock et al. , 2018 ; Choi et al. , 2018 ) , which are capable of producing high resolution images at very high quality . It is known that the earlier scales in such models are responsible for generating the global composition of the image , while the deeper scales are responsible for more local attributes ( Karras et al. , 2019a ; Yang et al. , 2019 ; Härkönen et al. , 2020 ) . Here , we distil this common knowledge and show how meaningful directions can be detected in each level , and how these architectures allow transferring attributes between images . 2 USER-SPECIFIED GEOMETRIC TRANSFORMATIONS . Most modern generator architectures map a latent code vector z ∈ Rd having no notion of spatial coordinates , into a two-dimensional output image . In some cases ( e.g. , BigGAN ) , different parts of z are processed differently . In others ( e.g. , BigGAN-deep ) , z is processed as a whole . However , in all cases , the first layer maps z ( or part of it ) into a tensor with low spatial resolution ( e.g. , 4×4×1536 in BigGAN 128 ) . This tensor is then processed by a sequence of convolutional layers that gradually increase its spatial resolution ( using fractional strides ) , until reaching the final image dimensions . Our key observation is that since the output of the first layer already has spatial coordinates , this layer has an important role in determining the coarse structure of the generated image . This suggests that if we were to apply a geometric transformation , like zoom or shift , on the output of the first layer , then we would obtain a similar effect to applying it directly on the generated image ( Fig . 2 ) . In fact , it may even allow slight semantic changes to take place due to the deeper layers that follow , which can compensate for the inability of the generator to generate the precise desired transformed image . As we now show , this observation can be used to find latent space directions corresponding to simple geometric transformations . 2.1 LINEAR TRAJECTORIES . Let us start with linear trajectories . Given a pre-trained generator G and some transformation T , our goal is to find a direction q in latent space such that G ( z + q ) ≈ T { G ( z ) } for every z . To this end , we define P to be the matrix corresponding to T in the resolution of the first layer ’ s output . Denoting the weights and biases of the first layer by W and b , respectively , our goal is therefore to bring1 W ( z + q ) + b as close as possible to P ( Wz + b ) . To guarantee that this holds on average over random draws of z , we formulate our problem as min q Ez∼pz [ ∥∥∥D ( W ( z + q ) + b− P ( Wz + b ) ) ∥∥∥2 ] , ( 1 ) where pz is the probability density function of z , and D is a diagonal matrix that can be used to assign different weights to different elements of the tensors . For example , if P corresponds to a horizontal shift of one element to the right , then we would not like to penalize for differences in the leftmost column of the shifted feature maps ( see Fig . 2 ) . In this case , we set the corresponding diagonal elements ofD to 0 and the rest to 1 . Assuming E [ z ] = 0 , as is the case in most frameworks , the objective in ( 1 ) simplifies to Ez∼pz [ ∥∥∥D ( ( I − P ) Wz ) ∥∥∥2 ] + ∥∥∥D ( Wq + ( I − P ) b ) ∥∥∥2 , ( 2 ) where I is the identity matrix . The first term in ( 2 ) is independent of q , and the second term is quadratic in q and is minimized by q = ( W TD2W ) −1 W TD2 ( P − I ) b . ( 3 ) We have thus obtained a closed form expression for the optimal linear direction corresponding to transformation P in terms of only the weightsW and b of the first layer . Figure 2 illustrates this framework in the context of the BigGAN model , in which the feature maps at the output of the first layer are 4× 4 . For translation , we use a matrix P that shifts the tensor by one element ( aiming at translating the output image by one fourth its size ) . For zoom-in , we use a matrix P that performs nearest-neighbor 2× up-sampling , and for zoom-out we use sub-sampling by 2× . For each such transformation , we can control the extent of the effect by multiplying the steering vector q by some α > 0 . Figure 1 ( top-left ) and Fig . 3 ( a ) show example results for zoom and shift with the BigGAN generator . As can be seen , this simple approach manages to produce pronounced effects , although not using optimization through the generator , as in ( Jahanian et al. , 2020 ) . Following ( Jahanian et al. , 2020 ) , we use an object detector to quantify our zoom and shift transformations . Figure 4 shows the distributions of areas and centers of object bounding boxes in the transformed images . As can be seen , our trajectories lead to similar effects to those of Jahanian et al . ( 2020 ) , despite being 104× faster to compute ( see Tab . 1 ) . Please refer to App . A.1 for details about the evaluation , and see additional results with BigGAN and with the DCGAN architecture of ( Miyato et al. , 2018 ) in App . A.3 .

Summary: The authors propose two new techniques that extract interpretable directions from latent spaces of pretrained GAN generators. Both techniques are very efficient and are shown to work with the state-of-the-art BigGAN models. Furthermore, the authors describe additional details of the method, like determining the transformation end-points, which are important for usage in the practical visual editing.