Text-Driven Image Manipulation via Semantic-Aware Knowledge Transfer

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have revolutionized a variety of fields due to its powerful ability to generate realistic and meaningful outputs . Recent works ( Jahanian et al. , 2019 ; He et al. , 2019 ; Goetschalckx et al. , 2019 ) have shown that deep generative models can capture real-world data distribution , and encode them into a semantically-rich latent space . Inspired by this , a lot of tasks draw their attention on the latent space manipulation , including image enhancement ( Ledig et al. , 2017 ; Yang et al. , 2021b ) , editing ( Shen et al. , 2020 ; Härkönen et al. , 2020 ; Patashnik et al. , 2021 ) , and discriminative tasks ( Nitzan et al. , 2021 ; Xu et al. , 2021 ) . With the tremendous success of deep generative models , facial attribute editing ( Yeh et al. , 2017 ; Liu et al. , 2019 ; He et al. , 2020 ; Dorta et al. , 2020 ) , aiming to edit the specific attributes of the target facial image , has become topical . As a special case of facial attribute editing , facial attribute transfer ( Xiao et al. , 2018 ; Lin et al. , 2018 ; Yin et al. , 2019 ; Choi et al. , 2018 ; 2020 ) uses knowledge from reference image as a condition to edit the corresponding attribute from the target image . In order to ensure that the manipulated facial image meets the requirements and interests , facial attribute transfer task tackles two challenges simultaneously : ( 1 ) editing relevance : the relevant attribute should be edited precisely according to the given condition ; and ( 2 ) keeping irrelevance : the irrelevant part ( e.g. , identify information , background , or other attributes ) should not be modified during attribute transfer . Due to strong entanglement of the attributes , meeting both requirements is an intractable task . For example , without fully disentanglement , transferring the “ smile ” attribute to the target facial image may cause that , another irrelevant but coupled attribute , e.g . “ cheek color ” attribute , would be changed during editing . In view of the above issues , recently a variety of methods explore the attribute disentanglement in two ways . Some methods ( He et al. , 2020 ; Kwak et al. , 2020 ) resort to the way of spatial attention detection , which disentangle the attribute by searching specific support region spatially and only manipulate the image in such a confined area . Obviously , these methods totally ignore the facial details beyond the support region when the edited attribute is a global attribute , such as “ smile ” or “ age ” . Meanwhile , other methods ( Shen et al. , 2020 ; Yang et al. , 2021a ; Patashnik et al. , 2021 ) pay attention to the latent space factorization through pre-trained GAN . These methods employ the high-level semantic information as guidance to manipulate image in latent spaces , which are more suitable to handle both global and local attribute editing . However , owing to over-coupled semantic features , these methods are hard to manipulate specific attribute without powerful supervision . To overcome the problems mentioned above , we explore the latent semantic space for disentangled attribute editing , and apply the discovered manipulation method to facial attribute transfer task . As for attribute editing task , in order to disentangle and edit the attribute specified by the text prompt , we design the directional latent mapping network , which leverages semantic direction consistency ( SDC ) loss to constrain the manipulation in the CLIP-space ( Radford et al. , 2021 ) . The key idea of the SDC loss is employing the change direction of semantic feature to estimate the latent manipulation . Furthermore , in order to apply this effective editing method to facial attribute transfer task , we propose a novel semantic-level facial attribute transfer method driven by text prompt , named as semantic directional decomposition network ( SDD-Net ) . The SDD-Net extracts and transfers the specific attribute without redundant information through attribute-manipulated semantic directional decomposition . Our contributions are summarized as follows : • We propose a novel method namely directional latent mapping network for facial attribute editing , which utilizes the semantic direction consistency regularization to ensure attribute disentanglement . • To further take advantage of semantic direction constrain , we propose a text-driven semantic directional decomposition network ( SDD-Net ) for semantic-level attribute transfer , by transferring the knowledge from the reference image to the target image . • Extensive experiments on CelebA-HQ ( Karras et al. , 2017 ) dataset show that our method achieves significant improvements over the state-of-art approaches . 2 RELATED WORK . Latent Space Manipulation . Recent studies ( Bau et al. , 2020 ; Goetschalckx et al. , 2019 ; Shen et al. , 2020 ) have shown that numerous GAN models can encode rich crucial information in the intermediate latent space , such asW , W+ ( Abdal et al. , 2019 ) , or StyleSpace S ( Wu et al. , 2021 ) . By learning to modify the intermediate latent code , generative models can transfer attributes from one face to another face ( Xiao et al. , 2018 ; Choi et al. , 2020 ) . To find a latent code that allows for meaningful manipulation , some methods try to learn an effective encoder network , which inverts a real image into latent space : encoder4editing ( e4e ) ( Tov et al. , 2021 ) method presents an encoder that is specifically designed for balancing distortion-editability tradeoff and a distortion-perception tradeoff within the StyleGAN latent space ; Stylespace ( Wu et al. , 2021 ) proposes a space of channelwise style parameters that disentangle attributes by controlling attribute-related style channels ; The pixel2style2pixel ( pSp ) ( Richardson et al. , 2021 ) method utilizes a novel encoder architecture that inverts a real image intoW+ space without optimization . Other methods mainly focus on finding such latent code modification approach as used to traverse latent space , result in the desired manipulation : AttGAN ( He et al. , 2019 ) applies attribute classification constraint to model the relation between the attributes and the latent representation ; InterfaceGAN ( Shen et al. , 2020 ) decouples some entangled semantic features with subspace projection ; GANspace ( Härkönen et al. , 2020 ) leverages principal component analysis to identify mainly direction in latent space ; L2M-GAN ( Yang et al. , 2021a ) imposes an orthogonality constraint to ensure disentanglement in latent space ; StyleCLIP ( Patashnik et al. , 2021 ) leverages CLIP models to guide the attributes manipulation by latent semantic matching . Different from the methods mentioned above , our method aligns the guidance direction , which is derived directly from text prompt , with the change direction of semantic features extracted by CLIP to guide the manipulation in the latent space . By constraining the consistency of these directions , our method can achieve disentangled attribute editing . CLIP-space + space Directional Latent Mapping Original face Edited face Figure 1 : Overview of our idea for directional latent mapping network . By enforcing the change direction of semantic feature to align with desired direction in CLIP-space , our method can achieve impressive disentangled image manipulation . Semantic-level Attribute Transfer . Semantic-level attribute transfer is a more challenging facial attribute editing task . Recent studies have attempted more detailed approaches for facial attribute transfer : StarGAN ( Choi et al. , 2018 ) applies cycle consistency to preserve identity , and uses classification loss to transfer between different domains . In addition , StarGANv2 ( Choi et al. , 2020 ) also could synthesize reference-guide images leveraging multiple domain translation . Some works are specialized for specific attributes : ExprGAN ( Ding et al. , 2018 ) proposes a model to learn the disentangled identity and expression representations explicitly for facial expression transfer . Besides , ERGAN ( Hu et al. , 2020 ) proposes a dual learning scheme to simultaneously learn two inverse manipulations for attribute transfer . Different from these works , our proposed method focuses not only on one attribute , but also on various global or local attributes . Given an explicit text prompt , our method can automatically extract the specific attribute from the reference , and transfer the knowledge to the target image . 3 METHODOLOGY . In the following section , we first provide the preliminaries and problem formulation . Then , we introduce our text-driven directional latent mapping network for disentangled facial attribute editing . Finally , we describe the text-driven semantic directional decomposition network ( SDD-Net ) for semantic-aware facial attribute transfer . 3.1 PRELIMINARIES . StyleGAN . The StyleGAN ( Karras et al. , 2019 ; 2020 ) generator consists of two main components : mapping network and synthesis network . The former translates latent code z to latent code w , which is in semantic-rich latent space W , and the latter utilizes latent code w to synthesize final images through different layers . Due to the rich information inside , the latent code w can control the multigranularity semantic features of the synthetic image , which is utilized for effective facial attribute transfer in semantic-level . In this paper , we leverage the pre-trained synthesis network as our image generator . StyleCLIP . The StyleCLIP ( Patashnik et al. , 2021 ) method first combines StyleGAN and CLIP as a strong tool for text-driven image editing . The key idea of StyleCLIP is to leverage the CLIP as latent manipulation guidance . By mapping multi-modal inputs to the CLIP-space , StyleCLIP could ensure that the synthetic image matches with the text prompt in semantic-level . The objective is given by : Lstyleclip = DCLIP ( G ( w ) , text ) , ( 1 ) where DCLIP is the cosine distance metric in CLIP-space , and G is the pre-trained StyleGAN generator . 3.2 PROBLEM FORMULATION . Our method leverages two latent space : W+ space and CLIP-space , to transfer semantic-level facial attributes . For better expression , let X and Y denote the set of images and semantic features in CLIP-space , respectively . The image x ∈ X is inverted into corresponding latent code w ∈ W+ . t denotes the input text prompt , with corresponding semantic feature yt . Meanwhile , the parameters of pre-trained StyleGANv2 generator G are frozen during training . Given the text prompt t , the goal of facial attribute transfer model is to train a latent mapping network , which translates w\y to ŵ\ŷ , and synthesizes the edited image x̂ that meets the specific requirements . Our basic editing model can be formally defined as x̂ = G ( M ( x , t ) ) , where M is the manipulation network . Hence given the reference image xref ∈ X , our semantic-aware attribute transfer model can be formally defined as x̂ = G ( M ( x , xref , t ) ) . 3.3 DIRECTIONAL LATENT MAPPING . Only focusing on matching ŷ with yt may cause irrelevant attribute changed . Therefore , we enforce the mapping network to focus on the change direction of semantic features in CLIP-space , and illustrate this idea in Figure 1 . The details of the proposed directional latent mapping network is described as follows : Architecture . It has been shown that the different layers of synthesis network , which correspends to different parts of the latent code , control different granularity of semantic feature . In order to better exploit this property , we design the directional latent mapping network , and depict it in Figure 2 . To a great extent , the degree of attributes disentanglement depends on the disentanglement of latent features . Therefore , we split the layers into g groups , instead three ( coarse , medium , and fine ) , with g fully connected mapping networks , one for each group . The divided part of the latent code can be denoted as w = [ w1 , w2 , ... , wg ] , so the mapping network is defined by : M ( w ) = [ M1 ( w1 ) , M2 ( w2 ) , ... , Mg ( wg ) ] , ( 2 ) where Mi is the i-th mapping network , and [ · , · ] means concat operation . Then we use skipconnection operation to obtain the final manipulated latent code ŵ = w + αM ( w ) , and feed it to the pre-trained StyleGANv2 generator G to get the final manipulated facial image x̂ = G ( ŵ ) . Training Objective . The edited desired attribute is determined by the textual prompt t. Without paired training data , the mapping network could not correctly manipulate the latent code . In order to obtain extra powerful supervision , we use CLIP model to effectively extract semantic features ( attributes ) y of the corresponding images and text : yi ; ŷi = EI ( x ; x̂ ) , yt = ET ( t ) , ( 3 ) where E denotes the pre-trained multi-modal feature extractor integrated in CLIP , yi and ŷi denotes the extracted semantic features of x and x̂ , respectively . yt is the desired semantic feature extracted from textual prompt t. In the latent CLIP-space , simply optimizing the matching degree between ŷi and yt may cause irrelevant attribute to be changed . Therefore , rather than optimizing the matching degree to the utmost extent , we enforce the change direction between y and ŷi to align with the yt , and propose the semantic direction consistency ( SDC ) loss . The SDC loss is given by : ~I = ŷi − yi , ~T = yt , ( 4 ) LSDC = 1− S ( ~I , ~T ) , where S ( · , · ) is the similarity measurement . In this paper , we use the effective cosine similarity as the measurement . The above objective can guide the mapping network to manipulate the latent code along the direction of textual prompt . In order to preserve the irrelevant parts , we use the following identity loss : LID = 1− 〈R ( G ( w ) ) , R ( G ( ŵ ) ) 〉 , ( 5 ) where R is a pre-trained ArcFace ( Deng et al. , 2019 ) network for face recognition , and 〈· , ·〉 computes the cosine similarity between its two arguments . Meanwhile , we use L2 distance in W+ space to control the degree of manipulation . Then , the whole training objective for directional latent mapping network is denoted as : argmin w∈W+ λSDCLSDC ( w , t ) + λL2||M ( w ) ||2 + λIDLID ( w ) . ( 6 )

Summary: This paper proposes a new loss function for unsupervised facial attribute editing and transfer. Specifically, a latent mapping network is trained by optimizing the similarity/distance between the generated and desired image in the CLIP latent space. Experiments show some comparisons and ablations for the "smile" attribute.