Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation

1 INTRODUCTION . Semantic image editing is the task of transforming a source image to a target image while modifying desired semantic attributes , e.g. , to make an image taken during summer look like it was captured in winter . The ability to semantically edit images is useful for various real-world tasks , including artistic visualization , design , photo enhancement , and targeted data augmentation . To this end , semantic image editing has two primary goals : ( i ) providing continuous manipulation of multiple attributes simultaneously and ( ii ) maintaining the original image ’ s identity as much as possible while ensuring photo-realism . Existing GAN-based approaches for semantic image editing can be categorized roughly into two groups : ( i ) image-space editing methods directly transform one image to another across domains ( Choi et al. , 2018 ; 2020 ; Isola et al. , 2017 ; Lee et al. , 2020 ; Wu et al. , 2019 ; Zhu et al. , 2017a ; b ) , usually using variants of generative adversarial nets ( GANs ) ( Goodfellow et al. , 2014 ) . These approaches often have high computational cost , and they primarily focus on binary attribute ( on/off ) changes , rather than providing continuous attribute editing abilities . ( ii ) latent-space editing methods focus on discovering latent variable manipulations that permit continuous semantic image edits . The chosen latent space is most often the latent space of GANs . Both unsupervised and ( self- ) supervised latent space editing methods have been proposed . Unsupervised latent-space editing methods ( Härkönen et al. , 2020 ; Voynov & Babenko , 2020 ) are often less effective at providing semantically meaningful directions and all too often change image identity during an edit . Current ( self- ) supervised methods ( Jahanian et al. , 2019 ; Plumerault et al. , 2020 ) are limited to geometric edits such as rotation and scale . To our knowledge , only one supervised approach has been proposed ( Shen et al. , 2019 ) – developed to discover semantic latent-space directions for binary attributes . As we show , this method suffers from entangled attributes and often does not preserve image identity during manipulation . Contributions . We propose a latent-space editing framework for semantic image manipulation that fulfills the aforementioned primary goals . Specifically , we use a GAN and employ a joint sampling strategy trained to edit multiple attributes simultaneously . To disentangle attribute transformations Original Inversion + Night + Winter + Clouds + Sunny B as el in e O ur s in the latent space of GANs , we integrate a regressor to predict the attributes that an image exhibits . The regressor also permits precise control of the manipulation degree and is easily extended to multiple attributes simultaneously . In addition , we incorporate a perceptual loss ( Li et al. , 2019 ) and an adversarial loss that helps preserve image identity and photo-realism during manipulation . We compare our method to several popular frameworks , from existing image-to-image translation methods ( Choi et al. , 2020 ; Wu et al. , 2019 ; Zhu et al. , 2017a ) to latent space transformation-based approaches ( Shen et al. , 2019 ; Voynov & Babenko , 2020 ) . We mention that prior work primarily uses qualitative evaluation like the one in Fig . 1 . In contrast , we propose a quantitative evaluation to measure controllability . Both qualitative and quantitative results provide evidence that our approach outperforms prior work in terms of quality of the semantic image manipulation while maintaining image identity . 2 RELATED WORK . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have significantly improved realistic image generation in recent years ( Brock et al. , 2018 ; Jolicoeur-Martineau , 2019 ; Karras et al. , 2017 ; 2019a ; b ; Park et al. , 2019 ; Zhang et al. , 2018 ) . For this , a GAN formulates a 2-player non-cooperative game between two deep nets : ( i ) a generator that produces an image given a random noise vector in the latent space , sampled from a known prior distribution , usually a normal or a uniform distribution ; ( ii ) a discriminator whose input is both synthetic and real data , which is to be differentiated . Semantic image editing seeks to automate image manipulation of semantics . Encouraged by the success of deep nets , recent works have applied deep learning methods for semantic image editing tasks such as style transfer ( Li et al. , 2017 ; Luan et al. , 2017 ) , image-to-image translation ( Choi et al. , 2018 ; 2020 ; Isola et al. , 2017 ; Lee et al. , 2020 ; Wang et al. , 2018 ; Wu et al. , 2019 ; Zhu et al. , 2017a ) and discovering semantically meaningful directions in a GAN latent space ( Härkönen et al. , 2020 ; Jahanian et al. , 2019 ; Plumerault et al. , 2020 ; Shen et al. , 2019 ; Voynov & Babenko , 2020 ) . Note that our task is an extended version of semantic image editing that requires more comprehensive control to satisfy user-desired operations . Therefore , most aforementioned approaches do not meet the requirements . Nonetheless , we categorize the approaches most relevant to our task into two groups : ( i ) image editing via manipulation in image space , and ( ii ) image editing via latent space navigation . Image-space editing using GANs directly manipulates an image for targeted editing . Early work ( Isola et al. , 2017 ) used GANs to implement semantic translations between two image domains with paired data , e.g. , day to night . Follow-up work focused on multi-modal ( Bhattarai & Kim , 2020 ; Choi et al. , 2018 ; 2020 ; Lee et al. , 2020 ; Wang et al. , 2018 ; Wu et al. , 2019 ) and unpaired image domain translation ( Zhu et al. , 2017a ) . In this case , they primarily consider binary ( on/off ) attribute changes , regardless of whether the process is dynamic , for example , day to night . Latent-space editing via GANs has received an increasing amount of recent interest . Most prior work focused on identifying semantically meaningful directions in the latent space of GANs , so that shifting latent vectors towards these directions achieves the desired image manipulation . Recent papers found semantics in the latent space of GANs , such as color transformations and camera movements ( Jahanian et al. , 2019 ) , or face attribute changes ( Shen et al. , 2019 ) , such as smile . Other work considered unsupervised methods ( Härkönen et al. , 2020 ; Voynov & Babenko , 2020 ) to discover interpretable latent space directions . However , additional challenges inherent to unsupervised manipulation of the latent space arise , and have not been addressed in prior work , e.g. , direction quality with regard to degree control and image identity preservation . 3 METHOD . 3.1 PROBLEM STATEMENT . We consider controllable semantic image editing via latent space navigation in GANs . We begin with a fixed GAN model that consists of a generator G and a discriminator D. The input of G is a latent vector z ∈ Rm from a latent space Z . Here , m is the dimensionality of the latent space . GivenN attributes , we aim to discover semantically meaningful latent-space GAN directions , T = { d1 , . . . , dN } , to manipulate the attributes of synthetic images G ( z ) with an assigned step size ε = { ε1 , . . . , εN } , where di ∈ Rm for all i ∈ { 1 , . . . , N } . In the end , each attribute of an edited image can be changed with the corresponding degree ε fromα = { α1 , . . . , αN } , the original attributes of G ( z ) predicted by a regressor R . 3.2 PROPOSED APPROACH . We provide an overview of our method in Fig . 2 ( a ) , and further illustrate how our approach differs from prior work in Fig . 2 ( b-d ) . Overview . As shown in Fig . 2 ( a ) , we employ a GAN model consisting of a generator G and a discriminator D ( not shown due to limited space ) , as well as a regressor R , all of which are pretrained . Our goal is to find latent directions T that provide attribute specific image edits . At each training step , G takes z and an edited latent vector referred to as z′ . We follow prior work which suggests that direction vectors in latent space of GANs permit image editing ( Jahanian et al. , 2019 ) . Formally , given a transformation degree ε , we obtain the latent-space edit as z′ = z + Tε = z + ∑N i=1 εidi . G provides the recovered and edited images G ( z ) and G ( z ′ ) , which are separately processed by the regressor R to predict attribute values . The original attributes of G ( z ) are α = R ( G ( z ) ) . The ( pseudo- ) ground truth and predicted attribute of G ( z′ ) are α′ and α̂′ , respectively , where α′ = α+ ε and α̂′ = R ( G ( z′ ) ) . Intuitively , the goal of T is to transform z by adding semantically meaningful information such that the corresponding output image G ( z′ ) exhibits attribute changes ε from α . In practice , we normalize the range of attribute values to [ 0,1 ] , i.e. , both α and α′ ∈ [ 0,1 ] . During training , we maintain the unit range by controlling the given ε . Formally , ε is drawn from a distribution Dε uniform in [ −1 , 1 ] N while considering the constraint that 0 ≤ α+ ε ≤ 1 . Objective function . To find T we minimize the weighted objective : min T L , λ1Lreg + λ2Ldisc + λ3Lcontent . ( 1 ) Note , the objective is only used for optimizing T , while the other modules remain fixed . Hyperparameters λ1 , λ2 , and λ3 adjust the loss terms . The regression lossLreg assesses whether T performs the transformations indicated by ε . We express the regression loss via a binary cross entropy : Lreg = Ez∼Z , ε∼Dε [ −α̂′ logα′ − ( 1− α̂′ ) log ( 1−α′ ) ] . ( 2 ) Note thatα′ is from the distribution generated by z and ε . Please refer to the appendix C.1 for more details on this distribution . The second loss term Ldisc is computed using the discriminator D and measures the quality of the generated images , i.e. , Ldisc = Ez′∼Z′|z [ log ( 1−D ( G ( z′ ) ) ) ] . ( 3 ) Here we refer to the domain of z′ via Z ′ , which is conditioned on z . We write this using Z ′|z . Lastly , we use a content loss Lcontent , often also referred to as perceptual loss . It is designed to estimate the distance between two images and it is employed to maintain the image identity during the transformation . Specifically , we use the content loss term Lcontent = Ez∼Z , z′∼Z′|z ∑ i∈Dcontent ‖Fi ( G ( z′ ) ) − Fi ( G ( z ) ) ‖22 , ( 4 ) where Fi ( · ) denotes a feature function which extracts intermediate features from images . Dcontent indicates the layers of a pre-trained model which are used as features . We approximate the aforementioned expectations by empirical sampling . We defer algorithm details to Appendix A. Joint-distribution sampling and training . The regressor operates in a multi-label setting , i.e. , each data possesses multiple attributes . To ensure the disentanglement of attribute edits we sample synthetic images from the entire data distribution and find all transformations at once ( illustrated in Fig . 2 ( b , c ) right ) . In contrast , Shen et al . ( 2019 ) prepare training samples on the two opposing data subsets with regards to an attribute , e.g. , no clouds vs. many clouds , and find the directions with N one-vs-one classifiers ( sketched in Fig . 2 ( b , c ) left ) . Transformation module T . We study two types of transformations T : ( i ) global and ( ii ) local . A global transformation T refers to a semantic latent-space transformation identical for all z during inference . This is illustrated via parallel red dashed arrows in Fig . 2 ( d ) left . These global directions are commonly used ( Härkönen et al. , 2020 ; Jahanian et al. , 2019 ; Shen et al. , 2019 ; Viazovetskyi et al. , 2020 ) . However , a globally identical direction might not serve all data . In contrast , a local transformation T is a function of z which provides various directions for different z ( in Fig . 2 ( d ) right , shown via non-parallel red arrows ) . Formally , di = f iθ ( z ) , where f i θ is implemented via a deep net . In Sec . 4 we show that the local transformation T succeeds on failure cases of the global transformation .

Summary: This paper presents a new approach for the semantic image editing task by allowing the controllable transformation on the latent space. Authors proposed to integrate an attribute regression network for training the transformation functions. The local transformation T is learned from a simple MLP conditioned on the latent vector z. Two outputs of the regression module for the original latent vector z and the transformed one z+T*epsilon are used to minimize the cross-entropy loss. Experiments validate the effectiveness of the proposed method in terms of manipulation quality.