Using latent space regression to analyze and leverage compositionality in GANs

1 INTRODUCTION . Natural scenes are comprised of disparate parts and objects that humans can easily segment and interchange ( Biederman , 1987 ) . Recently , unconditional generative adversarial networks ( Karras et al. , 2017 ; 2019b ; a ; Radford et al. , 2015 ) have become capable of mimicking the complexity of natural images by learning a mapping from a latent space noise distribution to the image manifold . But how does this seemingly unstructured latent space produce a strikingly realistic and structured 1Dome image from : https : //www.technologyreview.com/2019/10/24/132370/mit-dome/ scene ? Here , we use a latent regressor to probe the latent space of a pretrained GAN , allowing us to uncover and manipulate the concepts that GANs learn about the world in an unsupervised manner . For example , given a church image , is it possible to swap one foreground tree for another one ? Given only parts of the building , can the missing portion be realistically filled ? To achieve these modifications , the generator must be compositional , i.e. , understanding discrete and separate representations of objects . We show that the pretrained generator – without any additional interventions – already represents these compositional properties in its latent code . Furthermore , these properties can be manipulated using a regression network that predicts the latent code of a given image . The pixels of this image then provide us with an intuitive interface to control and modify the latent code . Given the modified latent code , the network then applies image priors learned from the dataset , ensuring that the output is always a coherent scene regardless of inconsistencies in the input ( Fig . 1 ) . Our approach is simple – given a fixed pretrained generator , we train a regressor network to predict the latent code from an input image , while adding a masking modification to learn to handle missing pixels . To investigate the GAN ’ s ability to produce a globally coherent version of a scene , we hand the regressor a rough , incoherent template of the scene we desire , and use the two networks to convert it into a realistic image . Even though our regressor is never trained on these unrealistic templates , it projects the given image into a reasonable part of the latent space , which the generator maps onto the image manifold . This approach requires no labels or clustering of attributes ; all we need is a single example of approximately how we want the generated image to look . It only requires a forward pass of the regressor and generator , so there is low latency in obtaining the output image , unlike iterative optimization approaches that can require upwards of a minute to reconstruct an image . We use the regressor to investigate the compositional capabilities of pretrained GANs across different datasets . Using input images composed of different image parts ( “ collages ” ) , we leverage the generator to recombine this unrealistic content into a coherent image . This requires solving three tasks simultaneously – blending , alignment , and inpainting . We then investigate the GAN ’ s ability to independently vary localized portions of a given image . In summary , our contributions are : • We propose a latent regression model that learns to perform image reconstruction even in the case of incomplete images and missing pixels and show that the combination of regressor and generator forms a strong image prior . • Using the learned regressor , we show that the representation of the generator is already compositional in the latent code , without having to resort to intermediate layer activations . • There is no use of labelled attributes nor test-time optimization , so we can edit images based on a single example of the desired modification and reconstruct in real-time . • We use the regressor to probe what parts of a scene can vary independently , and investigate the difference between image mixing using the encoder and interpolation in latent space . • The same regressor setup can be used for a variety of other image editing applications , such as multimodal editing , scene completion , or dataset rebalancing . 2 RELATED WORK . Image Inversion . Given a target image , the GAN inversion problem aims to recover a latent code which best generates the target . Image inversion comes with a number of challenges , including 1 ) a complex optimization landscape and 2 ) the generator ’ s inability to reconstruct out-of-domain images . To relax the domain limitations of the generator , one possibility is to invert to a more flexible intermediate latent space ( Abdal et al. , 2019 ) , but this may allow the generator to become overly flexible and requires regularizers to ensure that the recovered latent code does not deviate too far from the latent manifold ( Pividori et al. , 2019 ; Zhu et al. , 2020 ; Wulff & Torralba , 2020 ) . An alternative to increasing the flexibility of the generator is to learn an ensemble of latent codes that approximate a target image when combined ( Gu et al. , 2019a ) . Due to challenging optimization , the quality of inversion depends on good initialization . A number of approaches use a hybrid of a latent regression network to provide an initial guess of the latent code with subsequent optimization of the latent code ( Bau et al. , 2019 ; Guan et al. , 2020 ) or the generator weights ( Zhu et al. , 2016 ; Bau et al. , 2020 ; Pan et al. , 2020 ) , while Huh et al . ( 2020 ) investigates gradient-free approaches for optimization . Besides inverting whole images , a different use case of image inversion through a generator is to complete partial scenes . When using optimization , this is achieved by only measuring the reconstruction loss on the known pixels ( Bora et al. , 2017 ; Gu et al. , 2019a ; Abdal et al. , 2020 ) , whereas in feed-forward methods , the missing region must be provided explicitly to the model . Rather than inverting to the latent code of a pretrained generator , one can train the generator and encoder jointly , based on modifications to the Variational Autoencoder ( Kingma & Welling , 2013 ) . Donahue et al . ( 2017 ) ; Donahue & Simonyan ( 2019 ) ; Dumoulin et al . ( 2016 ) use this setup to investigate the properties of latent representations learned during training , while Pidhorskyi et al . ( 2020 ) demonstrate a joint learning method that can achieve comparable image quality to recent GAN models . In our work , we investigate the emergent priors of a pretrained GAN using a masked latent regression network as an approximate image inverter . While such a regressor has lower reconstruction accuracy than optimization-based techniques , its lower latency allows us to investigate the learned priors in a computationally efficient way and makes real-time image editing incorporating such priors possible . Composition in Image Domains . To join segments of disparate image sources into one cohesive output , early works use hand-designed features , such as Laplacian pyramids for seamless blending ( Burt & Adelson , 1983 ) . Hays & Efros ( 2007 ) and Isola & Liu ( 2013 ) employ nearest-neighbor approaches for scene composition and completion . More recently , a number of deep network architectures have been developed for compositional tasks . For discriminative tasks , Tabernik et al . ( 2016 ) and Kortylewski et al . ( 2020 ) train CNNs with modified compositional architectures to understand model interpretability and reason about object occlusion in classification . For image synthesis , Mokady et al . ( 2019 ) and Press et al . ( 2020 ) use an autoencoder to encode , disentangle , and swap properties between two sets of images , while Shocher et al . ( 2020 ) mixes images in deep feature space while training the generator . Rather than creating models specifically for image composition or scene completion objectives , we investigate the ability of a pre-trained GAN to mix-and-match parts of its generated images . Related to our work , Besserve et al . ( 2018 ) estimates the modular structure of GANs by learning a casual model of latent representations , whereas we investigate the GAN ’ s compositional properties using image inversion . Due to the imprecise nature of image collages , compositing image parts also involves inpainting misaligned regions . However , in contrast to inpainting , in which regions have to be filled in otherwise globally consistent images ( Pathak et al. , 2016 ; Iizuka et al. , 2017 ; Yu et al. , 2018 ; Zeng et al. , 2020 ) , the composition problem involves correcting inconsistencies as well as filling in missing pixels . Image Editing . A recent topic of interest is editing images using generative models . A number of works propose linear attribute vector editing directions to perform image manipulation operations ( Goetschalckx et al. , 2019 ; Jahanian et al. , 2020 ; Shen et al. , 2019 ; Kingma & Dhariwal , 2018 ; Karras et al. , 2019a ; Radford et al. , 2015 ) . It is also possible to identify concepts learned in the generator ’ s intermediate layers by clustering intermediate representations , either using segmentation labels ( Bau et al. , 2018 ) or unsupervised clustering ( Collins et al. , 2020 ) , and change these representations to edit the desired concepts in the output image . Suzuki et al . ( 2018 ) use a spatial feature blending approach which mixes properties of target images in the intermediate feature space of a generator . On faces , editing can be achieved using a 3D parametric model to supervise the modification ( Tewari et al. , 2017 ; 2020 ) . In our work , we do not require clusters or concepts in intermediate layers to be defined a priori , nor do we need distinct input and output domains for approximate collages and real images , as in image translation tasks ( Zhu et al. , 2017 ; Almahairi et al. , 2018 ) . Unlike image manipulation using semantic maps ( Park et al. , 2019 ; Gu et al. , 2019b ) , our approach respects the style of the manipulation ( e.g . the specific color of the sky ) , which is lost in the semantic map representation . Our method shares commonalities with Richardson et al . ( 2020 ) , although we focus on investigating compositional properties rather than image-to-image translation . In our approach , we only require a single example of the approximate target property we want to modify and use regression into the latent space as a fast image prior to create a coherent output . This allows us to create edits that are not contingent on labelled concepts , and we do not need to modify or train the generator . 3 METHOD . 3.1 LATENT CODE RECOVERY IN GANS . GANs provide a mapping from a predetermined input distribution to a complex output distribution , e.g . from a standard normal Z to the image manifold X , but they are not easily invertible . In other words , given an image sample from the output distribution , it is not trivial to recover the sample from the input distribution that generated it . The image inversion objective aims to find the latent code z of GAN G that best recovers the desired target image x : z∗ = argmin z ( dist ( G ( z ) , x ) ) , ( 1 ) using some metric of image distance dist , such as pixel-wise L1 error or a metric based on deep features . This objective can be solved iteratively , using L-BFGS ( Liu & Nocedal , 1989 ) or other optimizers . However , iterative optimization is slow – it takes a large number of iterations to converge , is prone to local minima , and must be performed for each target image x independently . An alternative way of recovering the latent code z is to train a neural network to directly predict it from a given image x . In this case , the recovered latent code is simply the result of a feed-forward pass through a trained regressor network , z∗ = E ( x ) , where E can be used for any x ∈ X . To train the regressor ( or encoder ) network E , we use the latent encoder loss L = Ez∼N ( 0,1 ) , x=G ( z ) [ ||x−G ( E ( x ) ) ||22 + Lp ( x , G ( E ( x ) ) ) + Lz ( z , E ( x ) ) ] . ( 2 ) We sample z randomly from the latent distribution , and pass it through a pretrained generator G to obtain the target image x = G ( z ) . Between the target image x and the recovered image G ( E ( x ) ) , we use a mean square error loss to guide reconstruction and a perceptual loss Lp ( Zhang et al. , 2018 ) to recover details . Between the original latent code z and the recovered latent code E ( x ) , we use a latent recovery loss Lz . We use mean square error or a variant of cosine similarity for latent recovery , depending on the GAN ’ s input normalization . Additional details can be found in Supp . Sec . A.1.1 . Throughout this paper the generators are frozen , and we only optimize the weights of the encoder E. When using ProGAN ( Karras et al. , 2017 ) , we train the encoder network to directly invert to the latent code z . For StyleGAN ( Karras et al. , 2019b ) , we encode to an expandedW+ latent space ( Abdal et al. , 2019 ) . Once trained , the output of the latent regressor yields a latent code such that the reconstructed image looks perceptually similar to the target image .

Summary: In this paper, the authors propose a latent space regression method for analyzing and manipulating the latent space of pre-trained GAN models. Unlike existing optimization-based methods, an explicit latent code regressor is learned to map the input to the latent space. The authors apply this approach to several applications: image composition, attribute modification, image completion, and multimodal editing. They also present some analysis on the independence of semantic parts of an image.