On the "steerability" of generative adversarial networks

1 INTRODUCTION . The quality of deep generative models has increased dramatically over the past few years . When introduced in 2014 , Generative Adversarial Networks ( GANs ) could only synthesize MNIST digits and low-resolution grayscale faces ( Goodfellow et al. , 2014 ) . The most recent models , however , produce diverse high-resolution images that are often indistinguishable from natural photos ( Brock et al. , 2018 ; Karras et al. , 2018 ) . Science fiction has long dreamed of virtual realities filled of synthetic content as rich as , or richer , than the real world ( e.g. , The Matrix , Ready Player One ) . How close are we to this dream ? Traditional computer graphics can render photorealistic 3D scenes , but can not automatically generate detailed content . Generative models like GANs , in contrast , can create content from scratch , but we do not currently have tools for navigating the generated scenes in the same kind of way as you can walk through and interact with a 3D game engine . In this paper , we explore the degree to which you can navigate the visual world of a GAN . Figure 1 illustrates the kinds of transformations we explore . Consider the dog at the top-left . By moving in some direction of GAN latent space , can we hallucinate walking toward this dog ? As the figure indicates , and as we will show in this paper , the answer is yes . However , as we continue to zoom in , we quickly reach limits . Once the dog face fills the full frame , continuing to walk in this direction fails to increase the zoom . A similar effect occurs in the daisy example ( row 2 of Fig . 1 ) , where a direction in latent space moves the daisy up and down , but can not move it out of frame . We hypothesize that these limits are due to biases in the distribution of images on which the GAN is trained . For example , if the training dataset consists of centered dogs and daises , the same may be the case in GAN-generated images . Nonetheless , we find that some degree of transformation is possible . When and why can we achieve certain transformations but not others ? This paper seeks to quantify the degree to which we can achieve basic visual transformations by navigating in GAN latent space . In other words , are GANs “ steerable ” in latent space ? 1 We analyze the relationship between the data distribution on which the model is trained and the success in achieving these transformations . From our experiments , it is possible to shift the distribution of generated images to some degree , but we can not extrapolate entirely out of the dataset ’ s support . In particular , attributes can be shifted in proportion to the variability of that attribute in the training data . We further demonstrate an approach to increase model steerability by jointly optimizing the generator and latent direction , together with data augmentation on training images . One of the current criticisms of generative models is that they simply interpolate between datapoints , and fail to generate anything truly new , but our results add nuance to this story . It is possible to achieve distributional shift , but the ability to create realistic images from a modified distributions relies on sufficient diversity in the dataset along the dimension that we vary . Our main findings are : • A simple walk in the latent space of GANs achieves camera motion and color transformations in the output image space . These walks are learned in self-supervised manner without labeled attributes or distinct source and target images . • The linear walk is as effective as more complex non-linear walks , suggesting that the models learn to roughly linearize these operations without being explicitly trained to do so . • The extent of each transformation is limited , and we quantify a relationship between dataset variability and how much we can shift the model distribution . • The transformations are a general-purpose framework that work with different model architectures , e.g . BigGAN , StyleGAN , and DCGAN , and illustrate different disentanglement properties in their respective latent spaces . • Data augmentation improves steerability , as does jointly training the walk trajectory and the generator weights , which allows us to achieve larger transformation effects . 2 RELATED WORK . Latent space manipulations can be seen from several perspectives – how we achieve it , what limits it , and what it enables us to do . Our work addresses these three aspects together , and we briefly refer to each one in related work . Interpolations in latent space Traditional approaches to image editing with GAN latent spaces find linear directions that correspond to changes in labeled attributes , such as smile-vectors and gender-vectors for faces ( Radford et al. , 2015 ; Karras et al. , 2018 ) . However these manipulations are not exclusive to GANs ; in flow-based generative models , linearly interpolating between two encoded images allow one to edit a source image toward attributes of the target ( Kingma & Dhariwal , 2018 ) . Möllenhoff & Cremers ( 2019 ) proposes a modified GAN formulation by treating data 1We use the term “ steerable ” in analogy to the classic steerable filters of Freeman & Adelson ( 1991 ) . as directional k-currents , where moving along tangent planes naturally corresponds to interpretable manipulations . Upchurch et al . ( 2017 ) removes the generative model entirely and instead interpolates in the intermediate feature space of a pretrained classifier , again using feature mappings of source and target sets to determine an edit direction . Unlike these approaches , we learn our latentspace trajectories in a self-supervised manner without labeled attributes or distinct source and target images . Instead , we learn to approximate editing operations on individual source images . We find that linear trajectories in latent space can capture simple image manipulations , e.g. , zoom-vectors and shift-vectors , although we also obtain similar results using nonlinear trajectories . Dataset bias Biases from training data and network architecture both impact the generalization capacity of learned models ( Torralba & Efros , 2011 ; Geirhos et al. , 2018 ; Amini et al. ) . Dataset biases partly comes from human preferences in taking photos : we tend to take pictures in specific “ canonical ” views that are not fully representative of the entire visual world ( Mezuman & Weiss , 2012 ; Jahanian et al. , 2015 ) . Consequently , models trained with these datasets inherit their biases . This may result in models that misrepresent the given task – such as tendencies towards texture bias rather than shape bias on ImageNet classifiers ( Geirhos et al. , 2018 ) – and in turn limits their generalization performance on similar objectives ( Azulay & Weiss , 2018 ) . Our latent space trajectories transform the output corresponding to various image editing operations , but ultimately we are constrained by biases in the data and can not extrapolate arbitrarily far beyond the data ’ s support . Generative models for content creation The recent progress in generative models has opened interesting avenues for content creation ( Brock et al. , 2018 ; Karras et al. , 2018 ) , including applications that enable users to fine-tune the generated output ( Simon ; Zhu et al. , 2016 ; Bau et al. , 2018 ) . A by-product the current work is enable users to modify image properties by turning a single knob – the magnitude of the learned transformation in latent space . We further demonstrate that these image manipulations are not just a simple creativity tool ; they also provide us with a window into biases and generalization capacity of these models . Applications of latent space editing Image manipulations using generative models suggest several interesting downstream applications . For example , Denton et al . ( 2019 ) learns linear walks corresponding to various facial characteristics – they use these to measure biases in facial attribute detectors , whereas we study biases in the generative model that originate from training data . Shen et al . ( 2019 ) also assumes linear latent space trajectories and learns paths for face attribute editing according to semantic concepts such as age and expression , thus demonstrating disentanglement of the latent space . White ( 2016 ) suggests approaches to improve the learned manipulations , such as using spherical linear interpolations , resampling images to remove biases in attribute vectors , and using data augmentation as a synthetic attribute for variational autoencoders . Goetschalckx et al . ( 2019 ) applies a linear walk to achieve transformations corresponding to cognitive properties of an image such as memorability , aesthetics , and emotional valence . Unlike these works , we do not require an attribute detector or assessor function to learn the latent space trajectory , and therefore our loss function is based on image similarity between source and target images . In addition to linear walks , we explore using non-linear walks parametrized by neural networks for editing operations . 3 METHOD . Generative models such as GANs ( Goodfellow et al. , 2014 ) learn a mapping function G such that G : z → x . Here , z is the latent code drawn from a Gaussian density and x is an output , e.g. , an image . Our goal is to achieve transformations in the output space by moving in latent space , as shown in Fig . 2 . In general , this goal also captures the idea in equivariance , in which transformations in the input space result in equivalent transformations in the output space ( c.f . Hinton et al . ( 2011 ) ; Cohen et al . ( 2019 ) ; Lenc & Vedaldi ( 2015 ) ) . Objective We want to learn anN -dimensional vector representing the optimal path in latent space for a given transformation . The vector is multiplied with continuous parameter α which signifies the step size : large α values correspond to a greater degree of transformation , while small α values correspond to a lesser degree . Formally , we learn the walk w by minimizing the objective function : w∗ = arg min w Ez , α [ L ( G ( z+αw ) , edit ( G ( z ) , α ) ) ] . ( 1 ) Here , L measures the distance between the generated image after taking an α-step in the latent direction G ( z + αw ) and the target edit ( G ( z ) , α ) derived from the source image G ( z ) . We use L2 loss as our objective L , however we also obtain similar results when using the LPIPS perceptual image similarity metric ( Zhang et al. , 2018 ) ( see Appendix B.4.1 ) . Note that we can learn this walk in a fully self-supervised manner – we perform the edit ( · ) operation on an arbitrary generated image and subsequently the vector to minimize the objective . Let model ( α ) denote the optimized transformation vector w∗ with the step size α , defined as model ( α ) = G ( z + αw∗ ) . The previous setup assumes linear latent space walks , but we can also learn non-linear trajectories in which the walk direction depends on the current latent space position . For the non-linear walk , we learn a function , f∗ ( z ) , which corresponds to a small -step transformation edit ( G ( z ) , ) . To achieve bigger transformations , we apply f recursively , mimicking discrete Euler ODE approximations . Formally , for a fixed , we minimize L = Ez , n [ ||G ( fn ( z ) ) − edit ( G ( z ) , n ) ) || ] , ( 2 ) where fn ( · ) is an nth-order function composition f ( f ( f ( ... ) ) ) , and f ( z ) is parametrized with a neural network . We discuss further implementation details in Appendix A.4 . We use this function composition approach rather than the simpler setup of G ( z + αNN ( z ) ) because the latter learns to ignore the input z when α takes on continuous values , and is thus equivalent to the previous linear trajectory ( see Appendix A.3 for further details ) . Quantifying Steerability We further seek to quantify how well we can achieve desired image manipulations under each transformation . To this end , we compare the distribution of a given attribute , e.g. , “ luminance ” , in the dataset versus in images generated after walking in latent space . For color transformations , we consider the effect of increasing or decreasing the α coefficient corresponding to each color channel . To estimate the color distribution of model-generated images , we randomly sample N = 100 pixels per image both before and after taking a step in latent space . Then , we compute the pixel value for each channel , or the mean RGB value for luminance , and normalize the range between 0 and 1 . For zoom and shift transformations , we rely on an object detector which captures the central object in the image class . We use a MobileNet-SSD v1 ( Liu et al. , 2016 ) detector to estimate object bounding boxes , and average over image classes recognizable by the detector . For each successful detection , we take the highest probability bounding box corresponding to the desired class and use that to quantify the amount of transformation . For the zoom operation , we use the area of the bounding box normalized by the area of the total image . For shift in the X and Y directions , we take the center X and Y coordinates of the bounding box , and normalize by image width or height . Truncation parameters in GANs ( as used in Brock et al . ( 2018 ) ; Karras et al . ( 2018 ) ) trade off between the diversity of the generated images and sample quality . When comparing generated images to the dataset distribution , we use the largest possible truncation for the model and perform similar cropping and resizing of the dataset as done during model training ( see Brock et al . ( 2018 ) ) . When comparing the attributes of generated distributions under different α magnitudes to each other but not to the dataset , we reduce truncation to 0.5 to ensure better performance of the object detector . Reducing Transformation Limits Equations 1 and 2 learn a latent space walk assuming a pretrained generative model , thus keeping the model weights fixed . The previous approach allows us to understand the latent space organization and limitations in the model ’ s transformation capacity . To overcome these limits , we explore adding data augmentation by editing the training images with each corresponding transformation , and train the generative model with this augmented dataset . We also introduce a modified objective function that jointly optimizes the generator weights and a linear walk vector : G∗ , w∗ = arg min G , w ( Ledit + LGAN ) , ( 3 ) where the edit loss encourages low L2 error between learned transformation and target image : Ledit = L2 ( G ( z+αw ) − edit ( G ( z ) , α ) ) . ( 4 ) The GAN loss optimizes for discriminator error : LGAN = max D ( Ez , α [ D ( G ( z+αw ) ) ] − Ex , α [ D ( edit ( x , α ) ) ] ) , ( 5 ) where we draw images x from the training dataset and perform data augmentation by applying the edit operation on them . This optimization approach encourages the generator to organize its latent space so that the transformations lie along linear paths , and when combined with data augmentation , results in larger transformation ranges which we demonstrate in Sec . 4.4

Summary: This paper propose to study the generalization properties of GANs through interpolation. They first propose to learn a linear (and non-linear) interpolation in the latent space for a specific type of image transformation for example zoom, translation, rotation, luminance, etc... They show that linear interpolation in GANs can produce really realistic images along the path and enable to control and transform generated images to some extent. They then propose to measure to what extent the generated images can be transformed without "breaking".  Finally they show that the quality of the interpolation can be improved by learning the interpolation and generator jointly.