Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation

1 INTRODUCTION . Semantic image editing is the task of transforming a source image to a target image while modifying desired semantic attributes , e.g. , to make an image taken during summer look like it was captured in winter . The ability to semantically edit images is useful for various real-world tasks , including artistic visualization , design , photo enhancement , and targeted data augmentation . To this end , semantic image editing has two primary goals : ( i ) providing continuous manipulation of multiple attributes simultaneously and ( ii ) maintaining the original image ’ s identity as much as possible while ensuring photo-realism . Existing GAN-based approaches for semantic image editing can be categorized roughly into two groups : ( i ) image-space editing methods directly transform one image to another across domains ( Choi et al. , 2018 ; 2020 ; Isola et al. , 2017 ; Lee et al. , 2020 ; Wu et al. , 2019 ; Zhu et al. , 2017a ; b ) , usually using variants of generative adversarial nets ( GANs ) ( Goodfellow et al. , 2014 ) . These approaches often have high computational cost , and they primarily focus on binary attribute ( on/off ) changes , rather than providing continuous attribute editing abilities . ( ii ) latent-space editing methods focus on discovering latent variable manipulations that permit continuous semantic image edits . The chosen latent space is most often the latent space of GANs . Both unsupervised and ( self- ) supervised latent space editing methods have been proposed . Unsupervised latent-space editing methods ( Härkönen et al. , 2020 ; Voynov & Babenko , 2020 ) are often less effective at providing semantically meaningful directions and all too often change image identity during an edit . Current ( self- ) supervised methods ( Jahanian et al. , 2019 ; Plumerault et al. , 2020 ) are limited to geometric edits such as rotation and scale . To our knowledge , only one supervised approach has been proposed ( Shen et al. , 2019 ) – developed to discover semantic latent-space directions for binary attributes . As we show , this method suffers from entangled attributes and often does not preserve image identity during manipulation . Contributions . We propose a latent-space editing framework for semantic image manipulation that fulfills the aforementioned primary goals . Specifically , we use a GAN and employ a joint sampling strategy trained to edit multiple attributes simultaneously . To disentangle attribute transformations Original Inversion + Night + Winter + Clouds + Sunny B as el in e O ur s in the latent space of GANs , we integrate a regressor to predict the attributes that an image exhibits . The regressor also permits precise control of the manipulation degree and is easily extended to multiple attributes simultaneously . In addition , we incorporate a perceptual loss ( Li et al. , 2019 ) and an adversarial loss that helps preserve image identity and photo-realism during manipulation . We compare our method to several popular frameworks , from existing image-to-image translation methods ( Choi et al. , 2020 ; Wu et al. , 2019 ; Zhu et al. , 2017a ) to latent space transformation-based approaches ( Shen et al. , 2019 ; Voynov & Babenko , 2020 ) . We mention that prior work primarily uses qualitative evaluation like the one in Fig . 1 . In contrast , we propose a quantitative evaluation to measure controllability . Both qualitative and quantitative results provide evidence that our approach outperforms prior work in terms of quality of the semantic image manipulation while maintaining image identity . 2 RELATED WORK . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have significantly improved realistic image generation in recent years ( Brock et al. , 2018 ; Jolicoeur-Martineau , 2019 ; Karras et al. , 2017 ; 2019a ; b ; Park et al. , 2019 ; Zhang et al. , 2018 ) . For this , a GAN formulates a 2-player non-cooperative game between two deep nets : ( i ) a generator that produces an image given a random noise vector in the latent space , sampled from a known prior distribution , usually a normal or a uniform distribution ; ( ii ) a discriminator whose input is both synthetic and real data , which is to be differentiated . Semantic image editing seeks to automate image manipulation of semantics . Encouraged by the success of deep nets , recent works have applied deep learning methods for semantic image editing tasks such as style transfer ( Li et al. , 2017 ; Luan et al. , 2017 ) , image-to-image translation ( Choi et al. , 2018 ; 2020 ; Isola et al. , 2017 ; Lee et al. , 2020 ; Wang et al. , 2018 ; Wu et al. , 2019 ; Zhu et al. , 2017a ) and discovering semantically meaningful directions in a GAN latent space ( Härkönen et al. , 2020 ; Jahanian et al. , 2019 ; Plumerault et al. , 2020 ; Shen et al. , 2019 ; Voynov & Babenko , 2020 ) . Note that our task is an extended version of semantic image editing that requires more comprehensive control to satisfy user-desired operations . Therefore , most aforementioned approaches do not meet the requirements . Nonetheless , we categorize the approaches most relevant to our task into two groups : ( i ) image editing via manipulation in image space , and ( ii ) image editing via latent space navigation . Image-space editing using GANs directly manipulates an image for targeted editing . Early work ( Isola et al. , 2017 ) used GANs to implement semantic translations between two image domains with paired data , e.g. , day to night . Follow-up work focused on multi-modal ( Bhattarai & Kim , 2020 ; Choi et al. , 2018 ; 2020 ; Lee et al. , 2020 ; Wang et al. , 2018 ; Wu et al. , 2019 ) and unpaired image domain translation ( Zhu et al. , 2017a ) . In this case , they primarily consider binary ( on/off ) attribute changes , regardless of whether the process is dynamic , for example , day to night . Latent-space editing via GANs has received an increasing amount of recent interest . Most prior work focused on identifying semantically meaningful directions in the latent space of GANs , so that shifting latent vectors towards these directions achieves the desired image manipulation . Recent papers found semantics in the latent space of GANs , such as color transformations and camera movements ( Jahanian et al. , 2019 ) , or face attribute changes ( Shen et al. , 2019 ) , such as smile . Other work considered unsupervised methods ( Härkönen et al. , 2020 ; Voynov & Babenko , 2020 ) to discover interpretable latent space directions . However , additional challenges inherent to unsupervised manipulation of the latent space arise , and have not been addressed in prior work , e.g. , direction quality with regard to degree control and image identity preservation . 3 METHOD . 3.1 PROBLEM STATEMENT . We consider controllable semantic image editing via latent space navigation in GANs . We begin with a fixed GAN model that consists of a generator G and a discriminator D. The input of G is a latent vector z ∈ Rm from a latent space Z . Here , m is the dimensionality of the latent space . GivenN attributes , we aim to discover semantically meaningful latent-space GAN directions , T = { d1 , . . . , dN } , to manipulate the attributes of synthetic images G ( z ) with an assigned step size ε = { ε1 , . . . , εN } , where di ∈ Rm for all i ∈ { 1 , . . . , N } . In the end , each attribute of an edited image can be changed with the corresponding degree ε fromα = { α1 , . . . , αN } , the original attributes of G ( z ) predicted by a regressor R . 3.2 PROPOSED APPROACH . We provide an overview of our method in Fig . 2 ( a ) , and further illustrate how our approach differs from prior work in Fig . 2 ( b-d ) . Overview . As shown in Fig . 2 ( a ) , we employ a GAN model consisting of a generator G and a discriminator D ( not shown due to limited space ) , as well as a regressor R , all of which are pretrained . Our goal is to find latent directions T that provide attribute specific image edits . At each training step , G takes z and an edited latent vector referred to as z′ . We follow prior work which suggests that direction vectors in latent space of GANs permit image editing ( Jahanian et al. , 2019 ) . Formally , given a transformation degree ε , we obtain the latent-space edit as z′ = z + Tε = z + ∑N i=1 εidi . G provides the recovered and edited images G ( z ) and G ( z ′ ) , which are separately processed by the regressor R to predict attribute values . The original attributes of G ( z ) are α = R ( G ( z ) ) . The ( pseudo- ) ground truth and predicted attribute of G ( z′ ) are α′ and α̂′ , respectively , where α′ = α+ ε and α̂′ = R ( G ( z′ ) ) . Intuitively , the goal of T is to transform z by adding semantically meaningful information such that the corresponding output image G ( z′ ) exhibits attribute changes ε from α . In practice , we normalize the range of attribute values to [ 0,1 ] , i.e. , both α and α′ ∈ [ 0,1 ] . During training , we maintain the unit range by controlling the given ε . Formally , ε is drawn from a distribution Dε uniform in [ −1 , 1 ] N while considering the constraint that 0 ≤ α+ ε ≤ 1 . Objective function . To find T we minimize the weighted objective : min T L , λ1Lreg + λ2Ldisc + λ3Lcontent . ( 1 ) Note , the objective is only used for optimizing T , while the other modules remain fixed . Hyperparameters λ1 , λ2 , and λ3 adjust the loss terms . The regression lossLreg assesses whether T performs the transformations indicated by ε . We express the regression loss via a binary cross entropy : Lreg = Ez∼Z , ε∼Dε [ −α̂′ logα′ − ( 1− α̂′ ) log ( 1−α′ ) ] . ( 2 ) Note thatα′ is from the distribution generated by z and ε . Please refer to the appendix C.1 for more details on this distribution . The second loss term Ldisc is computed using the discriminator D and measures the quality of the generated images , i.e. , Ldisc = Ez′∼Z′|z [ log ( 1−D ( G ( z′ ) ) ) ] . ( 3 ) Here we refer to the domain of z′ via Z ′ , which is conditioned on z . We write this using Z ′|z . Lastly , we use a content loss Lcontent , often also referred to as perceptual loss . It is designed to estimate the distance between two images and it is employed to maintain the image identity during the transformation . Specifically , we use the content loss term Lcontent = Ez∼Z , z′∼Z′|z ∑ i∈Dcontent ‖Fi ( G ( z′ ) ) − Fi ( G ( z ) ) ‖22 , ( 4 ) where Fi ( · ) denotes a feature function which extracts intermediate features from images . Dcontent indicates the layers of a pre-trained model which are used as features . We approximate the aforementioned expectations by empirical sampling . We defer algorithm details to Appendix A. Joint-distribution sampling and training . The regressor operates in a multi-label setting , i.e. , each data possesses multiple attributes . To ensure the disentanglement of attribute edits we sample synthetic images from the entire data distribution and find all transformations at once ( illustrated in Fig . 2 ( b , c ) right ) . In contrast , Shen et al . ( 2019 ) prepare training samples on the two opposing data subsets with regards to an attribute , e.g. , no clouds vs. many clouds , and find the directions with N one-vs-one classifiers ( sketched in Fig . 2 ( b , c ) left ) . Transformation module T . We study two types of transformations T : ( i ) global and ( ii ) local . A global transformation T refers to a semantic latent-space transformation identical for all z during inference . This is illustrated via parallel red dashed arrows in Fig . 2 ( d ) left . These global directions are commonly used ( Härkönen et al. , 2020 ; Jahanian et al. , 2019 ; Shen et al. , 2019 ; Viazovetskyi et al. , 2020 ) . However , a globally identical direction might not serve all data . In contrast , a local transformation T is a function of z which provides various directions for different z ( in Fig . 2 ( d ) right , shown via non-parallel red arrows ) . Formally , di = f iθ ( z ) , where f i θ is implemented via a deep net . In Sec . 4 we show that the local transformation T succeeds on failure cases of the global transformation .

Summary: In this paper, the authors propose an image attribute editing method by manipulating the GAN latent vector. Specifically, this paper uses a pre-trained GAN to synthesize images, a pre-trained regressor to get the image attributes, and trains a network T to find meaningful latent-space directions. It then edits image attributes by modifying the input latent vector, described as z' = z + T(z)ε. The experimental results show that the proposed method performs better than other selected methods to some degree.