Text-Driven Image Manipulation via Semantic-Aware Knowledge Transfer

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have revolutionized a variety of fields due to its powerful ability to generate realistic and meaningful outputs . Recent works ( Jahanian et al. , 2019 ; He et al. , 2019 ; Goetschalckx et al. , 2019 ) have shown that deep generative models can capture real-world data distribution , and encode them into a semantically-rich latent space . Inspired by this , a lot of tasks draw their attention on the latent space manipulation , including image enhancement ( Ledig et al. , 2017 ; Yang et al. , 2021b ) , editing ( Shen et al. , 2020 ; Härkönen et al. , 2020 ; Patashnik et al. , 2021 ) , and discriminative tasks ( Nitzan et al. , 2021 ; Xu et al. , 2021 ) . With the tremendous success of deep generative models , facial attribute editing ( Yeh et al. , 2017 ; Liu et al. , 2019 ; He et al. , 2020 ; Dorta et al. , 2020 ) , aiming to edit the specific attributes of the target facial image , has become topical . As a special case of facial attribute editing , facial attribute transfer ( Xiao et al. , 2018 ; Lin et al. , 2018 ; Yin et al. , 2019 ; Choi et al. , 2018 ; 2020 ) uses knowledge from reference image as a condition to edit the corresponding attribute from the target image . In order to ensure that the manipulated facial image meets the requirements and interests , facial attribute transfer task tackles two challenges simultaneously : ( 1 ) editing relevance : the relevant attribute should be edited precisely according to the given condition ; and ( 2 ) keeping irrelevance : the irrelevant part ( e.g. , identify information , background , or other attributes ) should not be modified during attribute transfer . Due to strong entanglement of the attributes , meeting both requirements is an intractable task . For example , without fully disentanglement , transferring the “ smile ” attribute to the target facial image may cause that , another irrelevant but coupled attribute , e.g . “ cheek color ” attribute , would be changed during editing . In view of the above issues , recently a variety of methods explore the attribute disentanglement in two ways . Some methods ( He et al. , 2020 ; Kwak et al. , 2020 ) resort to the way of spatial attention detection , which disentangle the attribute by searching specific support region spatially and only manipulate the image in such a confined area . Obviously , these methods totally ignore the facial details beyond the support region when the edited attribute is a global attribute , such as “ smile ” or “ age ” . Meanwhile , other methods ( Shen et al. , 2020 ; Yang et al. , 2021a ; Patashnik et al. , 2021 ) pay attention to the latent space factorization through pre-trained GAN . These methods employ the high-level semantic information as guidance to manipulate image in latent spaces , which are more suitable to handle both global and local attribute editing . However , owing to over-coupled semantic features , these methods are hard to manipulate specific attribute without powerful supervision . To overcome the problems mentioned above , we explore the latent semantic space for disentangled attribute editing , and apply the discovered manipulation method to facial attribute transfer task . As for attribute editing task , in order to disentangle and edit the attribute specified by the text prompt , we design the directional latent mapping network , which leverages semantic direction consistency ( SDC ) loss to constrain the manipulation in the CLIP-space ( Radford et al. , 2021 ) . The key idea of the SDC loss is employing the change direction of semantic feature to estimate the latent manipulation . Furthermore , in order to apply this effective editing method to facial attribute transfer task , we propose a novel semantic-level facial attribute transfer method driven by text prompt , named as semantic directional decomposition network ( SDD-Net ) . The SDD-Net extracts and transfers the specific attribute without redundant information through attribute-manipulated semantic directional decomposition . Our contributions are summarized as follows : • We propose a novel method namely directional latent mapping network for facial attribute editing , which utilizes the semantic direction consistency regularization to ensure attribute disentanglement . • To further take advantage of semantic direction constrain , we propose a text-driven semantic directional decomposition network ( SDD-Net ) for semantic-level attribute transfer , by transferring the knowledge from the reference image to the target image . • Extensive experiments on CelebA-HQ ( Karras et al. , 2017 ) dataset show that our method achieves significant improvements over the state-of-art approaches . 2 RELATED WORK . Latent Space Manipulation . Recent studies ( Bau et al. , 2020 ; Goetschalckx et al. , 2019 ; Shen et al. , 2020 ) have shown that numerous GAN models can encode rich crucial information in the intermediate latent space , such asW , W+ ( Abdal et al. , 2019 ) , or StyleSpace S ( Wu et al. , 2021 ) . By learning to modify the intermediate latent code , generative models can transfer attributes from one face to another face ( Xiao et al. , 2018 ; Choi et al. , 2020 ) . To find a latent code that allows for meaningful manipulation , some methods try to learn an effective encoder network , which inverts a real image into latent space : encoder4editing ( e4e ) ( Tov et al. , 2021 ) method presents an encoder that is specifically designed for balancing distortion-editability tradeoff and a distortion-perception tradeoff within the StyleGAN latent space ; Stylespace ( Wu et al. , 2021 ) proposes a space of channelwise style parameters that disentangle attributes by controlling attribute-related style channels ; The pixel2style2pixel ( pSp ) ( Richardson et al. , 2021 ) method utilizes a novel encoder architecture that inverts a real image intoW+ space without optimization . Other methods mainly focus on finding such latent code modification approach as used to traverse latent space , result in the desired manipulation : AttGAN ( He et al. , 2019 ) applies attribute classification constraint to model the relation between the attributes and the latent representation ; InterfaceGAN ( Shen et al. , 2020 ) decouples some entangled semantic features with subspace projection ; GANspace ( Härkönen et al. , 2020 ) leverages principal component analysis to identify mainly direction in latent space ; L2M-GAN ( Yang et al. , 2021a ) imposes an orthogonality constraint to ensure disentanglement in latent space ; StyleCLIP ( Patashnik et al. , 2021 ) leverages CLIP models to guide the attributes manipulation by latent semantic matching . Different from the methods mentioned above , our method aligns the guidance direction , which is derived directly from text prompt , with the change direction of semantic features extracted by CLIP to guide the manipulation in the latent space . By constraining the consistency of these directions , our method can achieve disentangled attribute editing . CLIP-space + space Directional Latent Mapping Original face Edited face Figure 1 : Overview of our idea for directional latent mapping network . By enforcing the change direction of semantic feature to align with desired direction in CLIP-space , our method can achieve impressive disentangled image manipulation . Semantic-level Attribute Transfer . Semantic-level attribute transfer is a more challenging facial attribute editing task . Recent studies have attempted more detailed approaches for facial attribute transfer : StarGAN ( Choi et al. , 2018 ) applies cycle consistency to preserve identity , and uses classification loss to transfer between different domains . In addition , StarGANv2 ( Choi et al. , 2020 ) also could synthesize reference-guide images leveraging multiple domain translation . Some works are specialized for specific attributes : ExprGAN ( Ding et al. , 2018 ) proposes a model to learn the disentangled identity and expression representations explicitly for facial expression transfer . Besides , ERGAN ( Hu et al. , 2020 ) proposes a dual learning scheme to simultaneously learn two inverse manipulations for attribute transfer . Different from these works , our proposed method focuses not only on one attribute , but also on various global or local attributes . Given an explicit text prompt , our method can automatically extract the specific attribute from the reference , and transfer the knowledge to the target image . 3 METHODOLOGY . In the following section , we first provide the preliminaries and problem formulation . Then , we introduce our text-driven directional latent mapping network for disentangled facial attribute editing . Finally , we describe the text-driven semantic directional decomposition network ( SDD-Net ) for semantic-aware facial attribute transfer . 3.1 PRELIMINARIES . StyleGAN . The StyleGAN ( Karras et al. , 2019 ; 2020 ) generator consists of two main components : mapping network and synthesis network . The former translates latent code z to latent code w , which is in semantic-rich latent space W , and the latter utilizes latent code w to synthesize final images through different layers . Due to the rich information inside , the latent code w can control the multigranularity semantic features of the synthetic image , which is utilized for effective facial attribute transfer in semantic-level . In this paper , we leverage the pre-trained synthesis network as our image generator . StyleCLIP . The StyleCLIP ( Patashnik et al. , 2021 ) method first combines StyleGAN and CLIP as a strong tool for text-driven image editing . The key idea of StyleCLIP is to leverage the CLIP as latent manipulation guidance . By mapping multi-modal inputs to the CLIP-space , StyleCLIP could ensure that the synthetic image matches with the text prompt in semantic-level . The objective is given by : Lstyleclip = DCLIP ( G ( w ) , text ) , ( 1 ) where DCLIP is the cosine distance metric in CLIP-space , and G is the pre-trained StyleGAN generator . 3.2 PROBLEM FORMULATION . Our method leverages two latent space : W+ space and CLIP-space , to transfer semantic-level facial attributes . For better expression , let X and Y denote the set of images and semantic features in CLIP-space , respectively . The image x ∈ X is inverted into corresponding latent code w ∈ W+ . t denotes the input text prompt , with corresponding semantic feature yt . Meanwhile , the parameters of pre-trained StyleGANv2 generator G are frozen during training . Given the text prompt t , the goal of facial attribute transfer model is to train a latent mapping network , which translates w\y to ŵ\ŷ , and synthesizes the edited image x̂ that meets the specific requirements . Our basic editing model can be formally defined as x̂ = G ( M ( x , t ) ) , where M is the manipulation network . Hence given the reference image xref ∈ X , our semantic-aware attribute transfer model can be formally defined as x̂ = G ( M ( x , xref , t ) ) . 3.3 DIRECTIONAL LATENT MAPPING . Only focusing on matching ŷ with yt may cause irrelevant attribute changed . Therefore , we enforce the mapping network to focus on the change direction of semantic features in CLIP-space , and illustrate this idea in Figure 1 . The details of the proposed directional latent mapping network is described as follows : Architecture . It has been shown that the different layers of synthesis network , which correspends to different parts of the latent code , control different granularity of semantic feature . In order to better exploit this property , we design the directional latent mapping network , and depict it in Figure 2 . To a great extent , the degree of attributes disentanglement depends on the disentanglement of latent features . Therefore , we split the layers into g groups , instead three ( coarse , medium , and fine ) , with g fully connected mapping networks , one for each group . The divided part of the latent code can be denoted as w = [ w1 , w2 , ... , wg ] , so the mapping network is defined by : M ( w ) = [ M1 ( w1 ) , M2 ( w2 ) , ... , Mg ( wg ) ] , ( 2 ) where Mi is the i-th mapping network , and [ · , · ] means concat operation . Then we use skipconnection operation to obtain the final manipulated latent code ŵ = w + αM ( w ) , and feed it to the pre-trained StyleGANv2 generator G to get the final manipulated facial image x̂ = G ( ŵ ) . Training Objective . The edited desired attribute is determined by the textual prompt t. Without paired training data , the mapping network could not correctly manipulate the latent code . In order to obtain extra powerful supervision , we use CLIP model to effectively extract semantic features ( attributes ) y of the corresponding images and text : yi ; ŷi = EI ( x ; x̂ ) , yt = ET ( t ) , ( 3 ) where E denotes the pre-trained multi-modal feature extractor integrated in CLIP , yi and ŷi denotes the extracted semantic features of x and x̂ , respectively . yt is the desired semantic feature extracted from textual prompt t. In the latent CLIP-space , simply optimizing the matching degree between ŷi and yt may cause irrelevant attribute to be changed . Therefore , rather than optimizing the matching degree to the utmost extent , we enforce the change direction between y and ŷi to align with the yt , and propose the semantic direction consistency ( SDC ) loss . The SDC loss is given by : ~I = ŷi − yi , ~T = yt , ( 4 ) LSDC = 1− S ( ~I , ~T ) , where S ( · , · ) is the similarity measurement . In this paper , we use the effective cosine similarity as the measurement . The above objective can guide the mapping network to manipulate the latent code along the direction of textual prompt . In order to preserve the irrelevant parts , we use the following identity loss : LID = 1− 〈R ( G ( w ) ) , R ( G ( ŵ ) ) 〉 , ( 5 ) where R is a pre-trained ArcFace ( Deng et al. , 2019 ) network for face recognition , and 〈· , ·〉 computes the cosine similarity between its two arguments . Meanwhile , we use L2 distance in W+ space to control the degree of manipulation . Then , the whole training objective for directional latent mapping network is denoted as : argmin w∈W+ λSDCLSDC ( w , t ) + λL2||M ( w ) ||2 + λIDLID ( w ) . ( 6 )

Summary: The authors proposed a directional latent mapping network for facial attribute editing via text inputs. The directional latent mapping network could correctly edit relevant attributes while preserving irrelevant attributes via training with the semantic direction consistency (SDC) loss. This paved the way to a novel semantic directional decomposition network (SDD-Net) for text-driven facial attribute transfer: SDD-Net transfers semantic-aware attributes from reference images to a target, with the multi-modal approach guiding the process via text input descriptions. CelebA-HQ dataset was used to compare results with recent SOA methods.