GAN "Steerability" without optimization 

1 INTRODUCTION . Since their introduction by Goodfellow et al . ( 2014 ) , generative adversarial networks ( GANs ) have seen remarkable progress , with current models capable of generating samples of very high quality ( Brock et al. , 2018 ; Karras et al. , 2019a ; 2018 ; 2019b ) . In recent years , particular effort has been invested in constructing controllable models , which allow manipulating attributes of the generated images . These range from disentangled models for controlling e.g. , the hair color or gender of facial images ( Karras et al. , 2019a ; b ; Choi et al. , 2018 ) , to models that even allow specifying object relations ( Ashual & Wolf , 2019 ) . Most recently , it has been demonstrated that GANs trained without explicitly enforcing disentanglement , can also be easily “ steered ” ( Jahanian et al. , 2020 ; Plumerault et al. , 2020 ) . These methods can determine semantically meaningful linear directions in the latent space of a pre-trained GAN , which correspond to various different image transformations , such as zoom , horizontal/vertical shift , in-plane rotation , brightness , redness , blueness , etc . Interestingly , a walk in the revealed directions typically has a similar effect across all object categories that the GAN can generate , from animals to man-made objects . To detect such latent-space directions , the methods of Jahanian et al . ( 2020 ) and Plumerault et al . ( 2020 ) require a training procedure that limits them to transformations for which synthetic images can be produced for supervision ( e.g. , shift or zoom ) . Other works have recently presented unsupervised techniques for exposing meaningful directions ( Voynov & Babenko , 2020 ; Härkönen et al. , 2020 ; Peebles et al. , 2020 ) . These methods can go beyond simple user-specified transformations , but also require optimization or training of some sort ( e.g. , drawing random samples in latent space ) . In this paper , we show that for most popular generator architectures , it is possible to determine meaningful latent space trajectories directly from the generator ’ s weights without performing any kind of training or optimization . As illustrated in Fig . 1 , our approach supports both simple userdefined geometric transformations , such as shift and zoom , and unsupervised exploration of directions that typically reveals more complex controls , like the 3D pose of the camera or the blur of the background . We also discuss how to achieve attribute transfer between images , even across object categories ( see Fig . 1 ) , again without any training . We illustrate results mainly on BigGAN , which is class-conditional , but our trajectories are class-agnostic . Our approach is advantageous over existing methods in several respects . First , it is 104×-105× faster . Second , it seems to detect more semantic directions than other methods . And third , it allows explicitly accounting for dataset biases . First order dataset biases As pointed out by Jahanian et al . ( 2020 ) , dataset biases affect the extent to which a pre-trained generator can accommodate different transformations . For example , if all objects in the training set are centered , then no walk in latent space typically allows shifting an object too much without incurring degradation . This implies that a “ steering ” latent-space trajectory should have an end-point . Our nonlinear trajectories indeed possess such convergence points , which correspond to the maximally-transformed versions of the images at the beginning of the trajectories . Conveniently , the end-point can be computed in closed form , so that we can directly jump to the maximally-transformed image without performing a gradual walk . Second order dataset biases Dataset biases can also lead to coupling between transformations . For example , in many datasets zoomed-out objects can appear anywhere within the image , while zoomed-in objects are always centered . In this case , trying to apply a zoom transformation may also result in an undesired shift so as to center the enlarged object . Our unsupervised method allows controlling the extent to which transformation A comes on the expense of transformation B . 1.1 RELATED WORK . Walks in latent space Many works use walks in a GAN ’ s latent space to achieve various effects ( e.g. , ( Shen et al. , 2020 ; Radford et al. , 2015 ; Karras et al. , 2018 ; 2019b ; Denton et al. , 2019 ; Xiao et al. , 2018 ; Goetschalckx et al. , 2019 ) ) . The recent works of Jahanian et al . ( 2020 ) and Plumerault et al . ( 2020 ) specifically focus on determining trajectories which lead to simple user-specified transformations , by employing optimization through the ( pre-trained ) generator . Voynov & Babenko ( 2020 ) proposed an unsupervised approach for revealing dominant directions in latent space . This technique reveals more complex transformations , such as background blur and background removal , yet it also relies on optimization . Most recently , the work of Härkönen et al . ( 2020 ) studied unsupervised discovery of meaningful directions by using PCA on deep features of the generator . The method seeks linear directions in latent space that best map to those deep PCA vectors , and results in a set of non-orthogonal directions . Similarly to the other methods , it also requires a very demanding training procedure ( drawing random latent codes and regressing the latent directions ) , which can take a day for models like BigGAN . Nonlinear walks in latent space Linear latent-space trajectories may arrive at regions where the probability density is low . To avoid this , some methods proposed to replace the popular Gaussian latent space distribution by other priors ( Kilcher et al. , 2018 ) , or to optimize the generator together with the latent space ( Bojanowski et al. , 2018 ) . Others suggested to use nonlinear walks in latent space that avoid low-probability regions . For example , Jahanian et al . ( 2020 ) explored nonlinear trajectories parametrized by two-layer neural networks , while White ( 2016 ) proposed spherical paths for interpolating between two latent codes . Hierarchical GAN architectures Recently there is tendency towards hierarchical GAN architectures ( Karras et al. , 2018 ; 2019a ; Brock et al. , 2018 ; Choi et al. , 2018 ) , which are capable of producing high resolution images at very high quality . It is known that the earlier scales in such models are responsible for generating the global composition of the image , while the deeper scales are responsible for more local attributes ( Karras et al. , 2019a ; Yang et al. , 2019 ; Härkönen et al. , 2020 ) . Here , we distil this common knowledge and show how meaningful directions can be detected in each level , and how these architectures allow transferring attributes between images . 2 USER-SPECIFIED GEOMETRIC TRANSFORMATIONS . Most modern generator architectures map a latent code vector z ∈ Rd having no notion of spatial coordinates , into a two-dimensional output image . In some cases ( e.g. , BigGAN ) , different parts of z are processed differently . In others ( e.g. , BigGAN-deep ) , z is processed as a whole . However , in all cases , the first layer maps z ( or part of it ) into a tensor with low spatial resolution ( e.g. , 4×4×1536 in BigGAN 128 ) . This tensor is then processed by a sequence of convolutional layers that gradually increase its spatial resolution ( using fractional strides ) , until reaching the final image dimensions . Our key observation is that since the output of the first layer already has spatial coordinates , this layer has an important role in determining the coarse structure of the generated image . This suggests that if we were to apply a geometric transformation , like zoom or shift , on the output of the first layer , then we would obtain a similar effect to applying it directly on the generated image ( Fig . 2 ) . In fact , it may even allow slight semantic changes to take place due to the deeper layers that follow , which can compensate for the inability of the generator to generate the precise desired transformed image . As we now show , this observation can be used to find latent space directions corresponding to simple geometric transformations . 2.1 LINEAR TRAJECTORIES . Let us start with linear trajectories . Given a pre-trained generator G and some transformation T , our goal is to find a direction q in latent space such that G ( z + q ) ≈ T { G ( z ) } for every z . To this end , we define P to be the matrix corresponding to T in the resolution of the first layer ’ s output . Denoting the weights and biases of the first layer by W and b , respectively , our goal is therefore to bring1 W ( z + q ) + b as close as possible to P ( Wz + b ) . To guarantee that this holds on average over random draws of z , we formulate our problem as min q Ez∼pz [ ∥∥∥D ( W ( z + q ) + b− P ( Wz + b ) ) ∥∥∥2 ] , ( 1 ) where pz is the probability density function of z , and D is a diagonal matrix that can be used to assign different weights to different elements of the tensors . For example , if P corresponds to a horizontal shift of one element to the right , then we would not like to penalize for differences in the leftmost column of the shifted feature maps ( see Fig . 2 ) . In this case , we set the corresponding diagonal elements ofD to 0 and the rest to 1 . Assuming E [ z ] = 0 , as is the case in most frameworks , the objective in ( 1 ) simplifies to Ez∼pz [ ∥∥∥D ( ( I − P ) Wz ) ∥∥∥2 ] + ∥∥∥D ( Wq + ( I − P ) b ) ∥∥∥2 , ( 2 ) where I is the identity matrix . The first term in ( 2 ) is independent of q , and the second term is quadratic in q and is minimized by q = ( W TD2W ) −1 W TD2 ( P − I ) b . ( 3 ) We have thus obtained a closed form expression for the optimal linear direction corresponding to transformation P in terms of only the weightsW and b of the first layer . Figure 2 illustrates this framework in the context of the BigGAN model , in which the feature maps at the output of the first layer are 4× 4 . For translation , we use a matrix P that shifts the tensor by one element ( aiming at translating the output image by one fourth its size ) . For zoom-in , we use a matrix P that performs nearest-neighbor 2× up-sampling , and for zoom-out we use sub-sampling by 2× . For each such transformation , we can control the extent of the effect by multiplying the steering vector q by some α > 0 . Figure 1 ( top-left ) and Fig . 3 ( a ) show example results for zoom and shift with the BigGAN generator . As can be seen , this simple approach manages to produce pronounced effects , although not using optimization through the generator , as in ( Jahanian et al. , 2020 ) . Following ( Jahanian et al. , 2020 ) , we use an object detector to quantify our zoom and shift transformations . Figure 4 shows the distributions of areas and centers of object bounding boxes in the transformed images . As can be seen , our trajectories lead to similar effects to those of Jahanian et al . ( 2020 ) , despite being 104× faster to compute ( see Tab . 1 ) . Please refer to App . A.1 for details about the evaluation , and see additional results with BigGAN and with the DCGAN architecture of ( Miyato et al. , 2018 ) in App . A.3 .

Summary: This paper studies transformations in GAN latent space that map to meaningful transformations in the generated data. The main contribution is to derive closed form methods for discovering latent transformations that correspond to 1) geometric changes and 2) changes that capture principle components of model variation. The paper also contributes new methods for nonlinear latent transformations, disentangled transformations, and an application to attribute transfer.