Year: 2021

Abstract:
Semantic-level facial attribute transfer is a special task to edit facial attribute, when reference images are viewed as conditions to control the image editing. In order to achieve better performance, semantic-level facial attribute transfer needs to fulfil two requirements: (1) specific attributes extracted from reference face should be precisely transferred to target face; (2) irrelevant information should be completely retained after transferring. Some existing methods locate and modify local support regions of facial images, which are not effective when editing global attributes; the other methods disentangle the latent code as different attributerelevant parts, which may transfer redundant knowledge to target faces. In this paper, we first propose a novel text-driven directional latent mapping network with semantic direction consistency (SDC) constrain to explore the latent semantic space for effective attribute editing, leveraging the semantic-aware knowledge of Contrastive Language-Image Pre-training (CLIP) model as guidance. This latent space manipulation strategy is designed to disentangle the facial attribute, removing the redundant knowledge in the transfer process. And on this basis, a novel attribute transfer method, named semantic directional decomposition network (SDD-Net), is proposed to achieve semantic-level facial attribute transfer by latent semantic direction decomposition, improving the interpretability and editability of our method. Extensive experiments on CelebA-HQ dataset show that our method achieves impressive performance over the state-of-the-art methods.

heading: 1 INTRODUCTION
text: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have revolutionized a variety of fields due to its powerful ability to generate realistic and meaningful outputs. Recent works (Jahanian et al., 2019; He et al., 2019; Goetschalckx et al., 2019) have shown that deep generative models can capture real-world data distribution, and encode them into a semantically-rich latent space. Inspired by this, a lot of tasks draw their attention on the latent space manipulation, including image enhancement (Ledig et al., 2017; Yang et al., 2021b), editing (Shen et al., 2020; Härkönen et al., 2020; Patashnik et al., 2021), and discriminative tasks (Nitzan et al., 2021; Xu et al., 2021).
With the tremendous success of deep generative models, facial attribute editing (Yeh et al., 2017; Liu et al., 2019; He et al., 2020; Dorta et al., 2020), aiming to edit the specific attributes of the target facial image, has become topical. As a special case of facial attribute editing, facial attribute transfer (Xiao et al., 2018; Lin et al., 2018; Yin et al., 2019; Choi et al., 2018; 2020) uses knowledge from reference image as a condition to edit the corresponding attribute from the target image. In order to ensure that the manipulated facial image meets the requirements and interests, facial attribute transfer task tackles two challenges simultaneously: (1) editing relevance: the relevant attribute should be edited precisely according to the given condition; and (2) keeping irrelevance: the irrelevant part (e.g., identify information, background, or other attributes) should not be modified during attribute transfer. Due to strong entanglement of the attributes, meeting both requirements is an intractable task. For example, without fully disentanglement, transferring the “smile” attribute to the target facial image may cause that, another irrelevant but coupled attribute, e.g. “cheek color” attribute, would be changed during editing.
In view of the above issues, recently a variety of methods explore the attribute disentanglement in two ways. Some methods (He et al., 2020; Kwak et al., 2020) resort to the way of spatial attention
detection, which disentangle the attribute by searching specific support region spatially and only manipulate the image in such a confined area. Obviously, these methods totally ignore the facial details beyond the support region when the edited attribute is a global attribute, such as “smile” or “age”. Meanwhile, other methods (Shen et al., 2020; Yang et al., 2021a; Patashnik et al., 2021) pay attention to the latent space factorization through pre-trained GAN. These methods employ the high-level semantic information as guidance to manipulate image in latent spaces, which are more suitable to handle both global and local attribute editing. However, owing to over-coupled semantic features, these methods are hard to manipulate specific attribute without powerful supervision.
To overcome the problems mentioned above, we explore the latent semantic space for disentangled attribute editing, and apply the discovered manipulation method to facial attribute transfer task. As for attribute editing task, in order to disentangle and edit the attribute specified by the text prompt, we design the directional latent mapping network, which leverages semantic direction consistency (SDC) loss to constrain the manipulation in the CLIP-space (Radford et al., 2021). The key idea of the SDC loss is employing the change direction of semantic feature to estimate the latent manipulation. Furthermore, in order to apply this effective editing method to facial attribute transfer task, we propose a novel semantic-level facial attribute transfer method driven by text prompt, named as semantic directional decomposition network (SDD-Net). The SDD-Net extracts and transfers the specific attribute without redundant information through attribute-manipulated semantic directional decomposition.
Our contributions are summarized as follows:
• We propose a novel method namely directional latent mapping network for facial attribute editing, which utilizes the semantic direction consistency regularization to ensure attribute disentanglement.
• To further take advantage of semantic direction constrain, we propose a text-driven semantic directional decomposition network (SDD-Net) for semantic-level attribute transfer, by transferring the knowledge from the reference image to the target image.
• Extensive experiments on CelebA-HQ (Karras et al., 2017) dataset show that our method achieves significant improvements over the state-of-art approaches.

heading: 2 RELATED WORK
text: Latent Space Manipulation. Recent studies (Bau et al., 2020; Goetschalckx et al., 2019; Shen et al., 2020) have shown that numerous GAN models can encode rich crucial information in the intermediate latent space, such asW ,W+ (Abdal et al., 2019), or StyleSpace S (Wu et al., 2021). By learning to modify the intermediate latent code, generative models can transfer attributes from one face to another face (Xiao et al., 2018; Choi et al., 2020). To find a latent code that allows for meaningful manipulation, some methods try to learn an effective encoder network, which inverts a real image into latent space: encoder4editing (e4e) (Tov et al., 2021) method presents an encoder that is specifically designed for balancing distortion-editability tradeoff and a distortion-perception tradeoff within the StyleGAN latent space; Stylespace (Wu et al., 2021) proposes a space of channelwise style parameters that disentangle attributes by controlling attribute-related style channels; The pixel2style2pixel (pSp) (Richardson et al., 2021) method utilizes a novel encoder architecture that inverts a real image intoW+ space without optimization. Other methods mainly focus on finding such latent code modification approach as used to traverse latent space, result in the desired manipulation: AttGAN (He et al., 2019) applies attribute classification constraint to model the relation between the attributes and the latent representation; InterfaceGAN (Shen et al., 2020) decouples some entangled semantic features with subspace projection; GANspace (Härkönen et al., 2020) leverages principal component analysis to identify mainly direction in latent space; L2M-GAN (Yang et al., 2021a) imposes an orthogonality constraint to ensure disentanglement in latent space; StyleCLIP (Patashnik et al., 2021) leverages CLIP models to guide the attributes manipulation by latent semantic matching.
Different from the methods mentioned above, our method aligns the guidance direction, which is derived directly from text prompt, with the change direction of semantic features extracted by CLIP to guide the manipulation in the latent space. By constraining the consistency of these directions, our method can achieve disentangled attribute editing.
CLIP-space + space
Directional Latent
Mapping Original face
Edited face
Figure 1: Overview of our idea for directional latent mapping network. By enforcing the change direction of semantic feature to align with desired direction in CLIP-space, our method can achieve impressive disentangled image manipulation.
Semantic-level Attribute Transfer. Semantic-level attribute transfer is a more challenging facial attribute editing task. Recent studies have attempted more detailed approaches for facial attribute transfer: StarGAN (Choi et al., 2018) applies cycle consistency to preserve identity, and uses classification loss to transfer between different domains. In addition, StarGANv2 (Choi et al., 2020) also could synthesize reference-guide images leveraging multiple domain translation. Some works are specialized for specific attributes: ExprGAN (Ding et al., 2018) proposes a model to learn the disentangled identity and expression representations explicitly for facial expression transfer. Besides, ERGAN (Hu et al., 2020) proposes a dual learning scheme to simultaneously learn two inverse manipulations for attribute transfer.
Different from these works, our proposed method focuses not only on one attribute, but also on various global or local attributes. Given an explicit text prompt, our method can automatically extract the specific attribute from the reference, and transfer the knowledge to the target image.

heading: 3 METHODOLOGY
text: In the following section, we first provide the preliminaries and problem formulation. Then, we introduce our text-driven directional latent mapping network for disentangled facial attribute editing. Finally, we describe the text-driven semantic directional decomposition network (SDD-Net) for semantic-aware facial attribute transfer.

heading: 3.1 PRELIMINARIES
text: StyleGAN. The StyleGAN (Karras et al., 2019; 2020) generator consists of two main components: mapping network and synthesis network. The former translates latent code z to latent code w, which is in semantic-rich latent space W , and the latter utilizes latent code w to synthesize final images through different layers. Due to the rich information inside, the latent code w can control the multigranularity semantic features of the synthetic image, which is utilized for effective facial attribute transfer in semantic-level. In this paper, we leverage the pre-trained synthesis network as our image generator.
StyleCLIP. The StyleCLIP (Patashnik et al., 2021) method first combines StyleGAN and CLIP as a strong tool for text-driven image editing. The key idea of StyleCLIP is to leverage the CLIP as latent manipulation guidance. By mapping multi-modal inputs to the CLIP-space, StyleCLIP could ensure that the synthetic image matches with the text prompt in semantic-level. The objective is given by:
Lstyleclip = DCLIP(G(w), text), (1)
where DCLIP is the cosine distance metric in CLIP-space, and G is the pre-trained StyleGAN generator.

heading: 3.2 PROBLEM FORMULATION
text: Our method leverages two latent space:W+ space and CLIP-space, to transfer semantic-level facial attributes. For better expression, let X and Y denote the set of images and semantic features in CLIP-space, respectively. The image x ∈ X is inverted into corresponding latent code w ∈ W+. t denotes the input text prompt, with corresponding semantic feature yt. Meanwhile, the parameters of pre-trained StyleGANv2 generator G are frozen during training.
Given the text prompt t, the goal of facial attribute transfer model is to train a latent mapping network, which translates w\y to ŵ\ŷ, and synthesizes the edited image x̂ that meets the specific requirements. Our basic editing model can be formally defined as x̂ = G(M(x, t)), where M is the manipulation network. Hence given the reference image xref ∈ X , our semantic-aware attribute transfer model can be formally defined as x̂ = G(M(x, xref , t)).

heading: 3.3 DIRECTIONAL LATENT MAPPING
text: Only focusing on matching ŷ with yt may cause irrelevant attribute changed. Therefore, we enforce the mapping network to focus on the change direction of semantic features in CLIP-space, and illustrate this idea in Figure 1. The details of the proposed directional latent mapping network is described as follows:
Architecture. It has been shown that the different layers of synthesis network, which correspends to different parts of the latent code, control different granularity of semantic feature. In order to better exploit this property, we design the directional latent mapping network, and depict it in Figure 2. To a great extent, the degree of attributes disentanglement depends on the disentanglement of latent features. Therefore, we split the layers into g groups, instead three (coarse, medium, and fine), with g fully connected mapping networks, one for each group. The divided part of the latent code can be denoted as w = [w1, w2, ..., wg], so the mapping network is defined by:
M(w) = [M1(w1),M2(w2), ...,Mg(wg)], (2) where Mi is the i-th mapping network, and [·, ·] means concat operation. Then we use skipconnection operation to obtain the final manipulated latent code ŵ = w + αM(w), and feed it to the pre-trained StyleGANv2 generator G to get the final manipulated facial image x̂ = G(ŵ).
Training Objective. The edited desired attribute is determined by the textual prompt t. Without paired training data, the mapping network could not correctly manipulate the latent code. In order to obtain extra powerful supervision, we use CLIP model to effectively extract semantic features (attributes) y of the corresponding images and text:
yi; ŷi = EI(x; x̂),
yt = ET (t), (3)
where E denotes the pre-trained multi-modal feature extractor integrated in CLIP, yi and ŷi denotes the extracted semantic features of x and x̂, respectively. yt is the desired semantic feature extracted from textual prompt t.
In the latent CLIP-space, simply optimizing the matching degree between ŷi and yt may cause irrelevant attribute to be changed. Therefore, rather than optimizing the matching degree to the utmost extent, we enforce the change direction between y and ŷi to align with the yt, and propose the semantic direction consistency (SDC) loss. The SDC loss is given by:
~I = ŷi − yi, ~T = yt, (4)
LSDC = 1− S(~I, ~T ),
where S(·, ·) is the similarity measurement. In this paper, we use the effective cosine similarity as the measurement.
The above objective can guide the mapping network to manipulate the latent code along the direction of textual prompt. In order to preserve the irrelevant parts, we use the following identity loss:
LID = 1− 〈R(G(w)), R(G(ŵ))〉 , (5)
where R is a pre-trained ArcFace (Deng et al., 2019) network for face recognition, and 〈·, ·〉 computes the cosine similarity between its two arguments. Meanwhile, we use L2 distance in W+ space to control the degree of manipulation. Then, the whole training objective for directional latent mapping network is denoted as:
argmin w∈W+
λSDCLSDC(w, t) + λL2||M(w)||2 + λIDLID(w). (6)

heading: 3.4 SEMANTIC DIRECTIONAL DECOMPOSITION
text: Inspired by the above method, we found that a semantic attribute translation can be represented as a directional vector in CLIP-space. Based on this, we leverage the same architecture of directional mapping network, and propose a facial attribute transfer method via semantic-aware knowledge transfer, named as semantic directional decomposition network (SDD-Net).
Given the textual prompt t, we want to transfer the attribute, extracted from the reference image xref , to the target image xtar. The yref and ytar are their corresponding semantic features respectively. And the semantic feature of textual prompt t is yt.
Due to the above direction-based manipulation, we can treat the facial attribute transfer problem as the attribute semantic feature projection problem. In the CLIP-space, a facial image can be treated as the composition of different semantic features. Therefore, we assume that different facial images can be converted to each other in semantic-level. In other words, one semantic feature can be translated to another semantic feature.
Under such an assumption, we first define the RT (reference-target) vector as: ~VRT = yref − ytar, which is the variance of two semantic features, and also is the direction of translation. We set the yt as the projection direction. In order to extract knowledge from the reference, we project RT vector ~VRT onto desired direction ~Vt = yt:
~Vp = ~VRT · ( ~Vt |Vt| )2 , (7)
where ~Vp is the projected vector. Then we add it to the target semantic feature ytar to get the final goal:
ygoal = ytar + β ~Vp, (8)
where β is the hyperparameter.
Training Objective. We use same architecture of directional latent mapping network to obtain the manipulated image x̂ = G(M(ŵ)), then match the semantic feature ŷ = EI(x̂) with ygoal in CLIP-space. As illustrated in Figure 3, the attribute transfer loss LAT is given by:
LAT = MSE(ŷ, ygoal), (9)
where MSE(·, ·) is the mean squared error. Meanwhile, we use the same L2 loss and identity loss to preserve the irrelevant parts. The whole training objective of SDD-Net is:
argmin w∈W+
λATLAT + λL2L2 + λIDLID. (10)

heading: 4 EXPERIMENTS
text: In this section, we first introduce the involved dataset and the implementation details. Then we will present the comparison results with several state-of-the-art facial attribute editing and facial attribute transfer methods to prove the effectiveness of our proposed method. Finally, the ablation studies will be presented to prove the effectiveness of our method.

heading: 4.1 DATASET
text: In order to achieve text-driven facial attribute editing and transfer, we choose the widely-used CelebA-HQ (Karras et al., 2017) dataset, which consists of 30,000 high quality facial images picked from the original CelebA (Liu et al., 2015) dataset. The size of each high quality image is 1024×1024. In the original dataset, each image has 40 attributes annotations inherited from the original CelebA. However in this work, we remove these annotations, and leverage CLIP model as powerful supervision. We also use the standard training, validation and test splits inherited from CelebA dataset.

heading: 4.2 IMPLEMENTATION DETAILS
text: In this subsection we provide the implementation details of our proposed networks. All images taken from the CelebA-HQ are inverted by e4e (Tov et al., 2021). We set the batch size and the number of total iterations to 5 and 50k respectively, during training. Our image editing is performed on StyleGANv2 pre-trained on FFHQ (Karras et al., 2019) dataset. We keep the StyleGANv2 generator fixed during training. The CLIP model is pre-trained on 400 million image-text pairs. We use the Vision Transformer (ViT) (Dosovitskiy et al., 2020) and a normal Transformer (Vaswani et al., 2017), which are integrated in CLIP model, as our image encoder and text encoder, respectively. Our directional latent mapping module is initialized using the pre-trained StyleGANv2, and trained
using Adam with the learning rate 5e-3. We set g = 9, α = 0.1, and β = 130. For facial attribute editing, the hyperparameters are empirically set as λSDC = 1, λL2 = 0.4, and λID = 0.02. For facial attribute transfer, the hyperparameters are empirically set as λAT = 1, λL2 = 0.2, and λID = 0.02. Our methods are trained on PyTorch with a single TITAN RTX GPU. We optimize training objective through gradient descent, by back-propagating the gradient through the pre-trained and fixed StyleGAN generator G and CLIP multi-modal encoder E.

heading: 4.3 BASELINE METHODS
text: Facial Attribute Editing Methods. We first compare our directional latent mapping network with the state-of-the-art facial attribute editing methods (i.e., L2M-GAN (Yang et al., 2021a), and StyleCLIP (Patashnik et al., 2021)) for the specific attribute: “smile”. Due to the requirements of highlevel semantic-aware knowledge, the smile attribute has become one of the most challenging global attributes. For L2M-GAN, we set the attribute domains as two (“smile” and “sad”), leveraging the domain label to guide the manipulation. For StyleCLIP, we use the latent mapper network as the baseline model, which is marked as StyleCLIP*.
Facial Attribute Transfer Methods. The StarGANv2 (Choi et al., 2020) learns to transform a source image reflecting the style of a given reference image. We compare our SDD-Net with StarGANv2 method on reference-guided image synthesis, which is a challenging facial attribute transfer task. Reference-guided image synthesis requires that various high-level attributes in the reference images, such as hairstyle, makeup, beard and expression, should be transferrd to the target images, while the irrelevant information such as pose and identity should be preserved. For fair comparison, we resize the images to 256×256 in reference-guided image synthesis.

heading: 4.4 RESULTS AND ANALYSIS
text: Facial Attribute Editing. The qualitative results are shown in Figure 4. After careful comparison, we have following observations: (1) The L2M-GAN performs well at disentangling attributes during editing. However, due to the strong constrain of orthogonal loss, the manipulated attribute is not obvious. (2) Although the StyleCLIP* method can make the best use of semantic knowledge of CLIP-space, the irrelevant attributes are changed in facial attribute editing without correct guidance direction. (3) Our directional latent mapping network manipulates the attribute correctly and naturally, which demonstrates that the proposed semantic directional consistency (SDC) loss could enforce the editing model to change specific attributes while preserving irrelevant parts. It also proves that there exists latent directions corresponding to different semantic properties in latent space. By manipulating latent code along such direction or its opposite direction, we can add or remove attributes.
Facial Attribute Transfer. Inspired by the above direction-based latent space manipulation for facial attribute transfer, we leverage this peculiarity for facial attribute transfer task.
We first execute our SDD-Net on the “smile” attribute for single attribute transfer, and input the corresponding text prompt to the model. The qualitative results are shown in Figure 5. We observe that when reference has the consistent attribute appointed by the prompt, the SDD-Net could correctly transfer the specific attribute to the target face with irrelevant information preserved, as shown in the top part. On the contrary, when the reference has the opposite attribute, our SDD-Net also could transfer opposite attribute to the target without extra guidance. By fully leveraging the knowledge of CLIP-space, our SDD-Net could find the semantic-aware latent direction in the latent space. The experiments show the excellent performance of our SDD-Net.
For the reference-guide image synthesis, we set the projected vector ~Vp is equal to the RT vector ~VRT . Figure 6 provides qualitative comparison of the results. We observe that StarGANv2 mothed synthesizes images with a same style code. However, the results show that StarGANv2 model manipulates images in a limited space. As a result, the attributes of the synthetic faces tend to be exactly alike. In addition, the expression attribute such as “smile” is ignored during transferring. Compared to StarGANv2, our SDD-Net transfers the multiple meaningful attributes to each of the target faces in semantic-level. Meanwhile, our method manipulates the image along the latent direction leveraging the knowledge of CLIP in latent space, which allows our method to synthesize realistic facial images, rather than simple style transfer.

heading: 4.5 ABLATION STUDY
text: We conduct a qualitative ablation study, as shown in Figure 7, and show the significance of identity loss. We observe that, when hyperparameter λID = 0.1, the ID loss hinders the attribute transfer. Then we experiment with λID = 0, the attribute could be correctly transferred, but the identity information is changed. To trade-off, we set the λID = 0.02.

heading: 5 CONCLUSIONS
text: In this paper, we first propose directional latent mapping network for text-driven facial attribute editing. By leveraging semantic direction consistency (SDC) loss, the directional latent mapping network could correctly edit relevant attribute while preserving irrelevant attributes. And on this basis, we propose semantic directional decomposition network (SDD-Net) for text-driven facial attribute transfer, which correctly transfers the semantic-aware attributes of reference image to the target image. Experiments show that our method achieves impressive performance on CelebA-HQ dataset.

Review summaries:
The authors proposed a directional latent mapping network for facial attribute editing via text inputs. The directional latent mapping network could correctly edit relevant attributes while preserving irrelevant attributes via training with the semantic direction consistency (SDC) loss. This paved the way to a novel semantic directional decomposition network (SDD-Net) for text-driven facial attribute transfer: SDD-Net transfers semantic-aware attributes from reference images to a target, with the multi-modal approach guiding the process via text input descriptions. CelebA-HQ dataset was used to compare results with recent SOA methods.

This paper proposes a new loss function for unsupervised facial attribute editing and transfer. Specifically, a latent mapping network is trained by optimizing the similarity/distance between the generated and desired image in the CLIP latent space. Experiments show some comparisons and ablations for the "smile" attribute.