Text-Driven Image Manipulation via Semantic-Aware Knowledge Transfer
Authors: Ziqi Zhang
Cheng Deng
Kun Wei
Xu Yang
Venue: ICLR 2022 Submitted (2022)

Abstract: Semantic-level facial attribute transfer is a special task to edit facial attribute, when reference images are viewed as conditions to control the image editing. In order to achieve better performance, semantic-level facial attribute transfer needs to fulfil two requirements: (1) specific attributes extracted from reference face should be precisely transferred to target face; (2) irrelevant information should be completely retained after transferring. Some existing methods locate and modify local support regions of facial images, which are not effective when editing global attributes; the other methods disentangle the latent code as different attribute-relevant parts, which may transfer redundant knowledge to target faces. In this paper, we first propose a novel text-driven directional latent mapping network with semantic direction consistency (SDC) constrain to explore the latent semantic space for effective attribute editing, leveraging the semantic-aware knowledge of Contrastive Language-Image Pre-training (CLIP) model as guidance. This latent space manipulation strategy is designed to disentangle the facial attribute, removing the redundant knowledge in the transfer process. And on this basis, a novel attribute transfer method, named semantic directional decomposition network (SDD-Net), is proposed to achieve semantic-level facial attribute transfer by latent semantic direction decomposition, improving the interpretability and editability of our method. Extensive experiments on CelebA-HQ dataset show that our method achieves impressive performance over the state-of-the-art methods.

Under review as a conference paper at ICLR 2022
TEXT-DRIVEN IMAGE MANIPULATION VIA SEMANTIC-AWARE KNOWLEDGE TRANSFER
Anonymous authors Paper under double-blind review
ABSTRACT
Semantic-level facial attribute transfer is a special task to edit facial attribute, when reference images are viewed as conditions to control the image editing. In order to achieve better performance, semantic-level facial attribute transfer needs to ful- fil two requirements: (1) specific attributes extracted from reference face should be precisely transferred to target face; (2) irrelevant information should be com- pletely retained after transferring. Some existing methods locate and modify lo- cal support regions of facial images, which are not effective when editing global attributes; the other methods disentangle the latent code as different attribute- relevant parts, which may transfer redundant knowledge target faces. In this paper, we first propose a novel text-driven directional latent mapping network with semantic direction consistency (SDC) constrain explore the latent seman- tic space for effective attribute editing, leveraging the semantic-aware knowledge of Contrastive Sanguagee-mmag Pre-training (CLIP) model as guidance. This la- tent space manipulation strategy is designed to disentangle the facial attribute, removing the redundant knowledge in the transfer process. And on this basis, a novel attribute transfer method, named semantic directional decomposition net- work (SDD-Net), is proposed to achieve semantic-level facial attribute transfer by latent semantic direction decomposition, improving the interpretability and ed- itability of our method. Extensive experiments on CelebA-HQ dataset show that our method achieves impressive performance over the ofate-ofline the-art methods.
INTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) have revolutionized a variety of fields due to its powerful ability to generate realistic and meaningful outputs. Recent works (Jahanian et al., 2019 He et al. 2019 Goetschalckx et al., 2019) have shown that deep generative models can capture real-world data distribution, and encode them into a semantically-rich latent space. Inspired by this, lot of tasks draw their attention on the latent space manipulation. including
image enhancement (Ledig et al. 2017 Yang et al.) 2021b), editing (Shen et al.. 2020 Härkönen al. 2020 Patashnik et al. 2021 and discriminative tasks NNizann et al. 2021| Xu et al. 2021).
With the tremendous success of deep generative models, facial attribute editing (Yeh et al. 2017]
Liu et al. 2019 He et al. 22200: Dorta et al. 2020), aiming edit the specific attributes of the target facial image, has become topical. As a special case of facial attribute editing, facial attribute transfer
(iaoo et al., 2018 Lin al. 2018 Yin al. 2019 Choi et al. 2018 2020 uses knowledge from reference image as a condition to edit the corresponding attribute from the target image. In order to ensure that the manipulated facial image meets the requirements and interests, facial attribute trans- fer task tackles two challenges simultaneously: (1) editing relevance: the relevant attribute should be edited precisely according to the given condition; and (2) keeping irrelevance: the irrelevant part (c.g., identify information, background, or other attributes) should not be modified during attribute transfer. Due to strong entanglement of the attributes, meeting both requirements is an intractable task. For example, without fully disentanglement, transferring the "smile" attribute to the target facial image may cause that, another irrelevant but coupled attribute, e.g. "cheek color" attribute, would be changed during editing.
In view of the above issues, recently a variety of methods explore the attribute disentanglement in two ways. Some methods (He et al. 2020: Kwak al. 2020) resort to the way of spatial attention

---

Under review as a conference paper at ICLR 2022
detection, which disentangle the attribute by searching specific support region spatially and only manipulate the image in such a confined area. Obviously, these methods totally ignore the facial details beyond the support region when the edited attribute is global attribute, such as "smile" or "age". Meanwhile, other methods (Shen et al., 2020, Yang et al. 2021a Patashnik et al. (021) pay attention the latent space factorization through pre-trained GAN. These methods employ the high-level semantic information as guidance to manipulate image in latent spaces, which are more suitable handle both global and local attribute editing. However, owing to over-coupled semantic features, these methods are hard to manipulate specific attribute without powerful supervision.
To overcome the problems mentioned above, we explore the latent semantic space for disentangled attribute editing, and apply the discovered manipulation method to facial attribute transfer task. As for attribute editing task, in order to disentangle and edit the attribute specified by the text prompt, we design the directional latent mapping network, which leverages semantic direction consistency (SDC) loss to constrain the manipulation in the CLIP--pace (Radford et al., (02T). The key idea of the SDC loss is employing the change direction of semantic feature to estimate the latent manipu- lation. Furthermore, in order to apply this effective editing method to facial attribute transfer task, propose novel semantic-level facial attribute transfer method driven by text prompt, named as semantic directional decomposition network (SDD-Net). The SDD-Net extracts and transfers the specific attribute without redundant information through sttribue-mmnipulle semantic directional decomposition.
Our contributions are summarized follows:
We propose novel method namely directional latent mapping network for facial attribute editing, which utilizes the semantic direction consistency regularization to ensure attribute disentanglement.
To further take advantage of semantic direction constrain, we propose a text-driven seman- tic directional decomposition network (SDD-Net) for semantic-lever attribute transfer, by transferring the knowledge from the reference image to the target image.
Extensive experiments on CelebA-HQ (Karras et al., (017) dataset show that method achieves significant improvements over the state-of-art approaches.
2 RELATED WORK
Latent Space Manipulation. Recent studies (Bau et al. 2020 Goetschalckx et al.) 2019 Shen al., 2020) have shown that numerous GAN models can encode rich crucial information in the intermediate latent space, such as W, W+ (Abdal et al. 2019), or StyleSpace S (Wu et al.) 2021 By learning to modify the intermediate latent code, generative models can transfer attributes from one face to another face (Xiao et al. 2018) Choi al. 2020). To find a latent code that allows for meaningful manipulation, some methods try to learn an effective encoder network, which inverts a encoder tradcoff and a distortion perrreioonn
real image into latent space: encoder4editing (e4e) (Tov et al. 2021) method presents that is specifically designed for balancing distortion-editabeee tradeoff within the StyleGAN latent space; Stylespace (Wu al., 2022) proposes a space of channel- wise style parameters that disentangle attributes by controlling attribute-related style channels; The pixel2style2pixel (pSp) (Richardson al. (02I)) method utilizes novel encoder architecture that inverts a real image into W+ space without optimization. Other methods mainly focus on finding such latent code modification approach as used traverse latent space, result in the desired manip- ulation: AttGAN (He et al. 2019) applies attribute classification constraint to model the relation be- tween the attributes and the latent representation; InterfaceGAN (Shen et al., (020)) decouples some entangled semantic features with subspace projection; GANspace (Härkönen et al., 2020) leverages principal component analysis to identify mainly direction in latent space; 1.2M-GAN (Yang et al.) 2021a) imposes orthogonality constraint to ensure disentanglement in latent space: StyleCLIP (Patashnik et al. 2021) leverages CLIP models to guide the attributes manipulation by latent seman- tic matching.
Different from the methods mentioned above, our method aligns the guidance direction, which is derived directly from text prompt, with the change direction of semantic features extracted by CLIP to guide the manipulation in the latent space. By constraining the consistency of these directions, method can achieve disentangled attribute editing.
2

---

Under review as a conference paper at ICLR 2022
CLIP-space
W+ space
"Smile"
Directional Latent Mapping
Original face
Edited face
Figure 1: Overview of our idea for directional latent mapping network. By enforcing the change
direction of semantic feature align with desired direction in CLIP-space, our method achieve impressive disentangled image manipulation.
Semantic-level Attribute Transfer. Semantic-level attribute transfer is more challenging facial attribute editing task. Recent studies have attempted more detailed approaches for facial attribute transfer: StarGAN (Choi al. 2018) applies cycle consistency preserve identity, and uses classi- fication loss to transfer between different domains. In addition, StarGANv2 (Chol et al.) 2020) also could synthesize reference-guide images leveraging multiple domain translation. Some works specialized for specific attributes: ExprGAN (Ding et al., (018)) proposes model to learn the dis- entangled identity and expression representations explicitly for facial expression transfer. Besides, ERGAN (Hu et al. 2020) proposes a dual learning scheme to simultancously learn two inverse manipulations for attribute transfer.
Different from these works, our proposed method focuses not only on one attribute, but also on various global or local attributes. Given an explicit text prompt, our method can automatically extract the specific attribute from the reference, and transfer the knowledge to the target image.
METHODOLOGY
In the following section, we first provide the preliminaries and problem formulation. Then, we introduce our text-driven directional latent mapping network for disentangled facial attribute edit- ing. Finally, we describe the text-driven semantic directional decomposition network (SDD-Net) for semantic-aware aware facial attribute transfer.
3.1 PRELIMINARIES
StyleGAN. The StyleGAN (Krrrs et al.) 2019 2020) generator consists of two main components: mapping network and synthesis network. The former translates latent code to latent code w, which is in semantic-rich latent space W, and the latter utilizes latent code to to synthesize final images through different layers. Due to the rich information inside, the latent code can control the multi- granularity semantic features of the synthetic image, which is utilized for effective facial attribute transfer in semantic-level. In this paper, we leverage the pre-trained synthesis network as our image generator.
StyleCLIP. The StyleCLIP (Patashnik et al.) 2021) method first combines StyleGAN and CLIP as strong tool for text-driven image editing. The key idea of StyleCLIP is to leverage the CLIP as latent manipulation guidance. By mapping multi-modal inputs the CLIP-space, StyleCLIP could ensure that the synthetic image matches with the text prompt in semantic-level. The objective is given by:
Cstyleclip = DCLIP CeLIP(G((e)) (G(w), text).
(1)
where DCLIP is the cosine distance metric CLIP-space, and the pre-trained StyleGAN gen- erator.
3

---

Under review as a conference paper at ICLR 2022
Inversion
StyleGAN Generator
Chaanel-WWis Mapping
Figure 2: The architecture of our directional latent mapping network. Given the text prompt. we manipulate the latent code inverted by the facial images. Then g group channel-wise mapping networks to manipulate the different parts latent code w respectively. Multiple loss functions are used to constrain the synthetic image to fulfil the requirements.
3.2 PROBLEM FORMULATION
Our method leverages latent space: W+ space and CLIP-space, transfer semantic-level facial attributes. For better expression, let X and y denote the set of images and semantic features in CLIP-space, respectively. The image x E X' inverted into corresponding latent code w E W++ denotes the input text prompt, with corresponding semantic feature yt- Meanwhile, the parameters of pre-trained StyleGANv2 generator G are frozen during training.
Given the text prompt I, the goal of facial attribute transfer model is to train latent mapping network, which translates w/y to w and synthesizes the edited image that meets the specific requirements. Our basic editing model can be formally defined as f G(M(x, t)). where M is the manipulation network. Hence given the reference image Iref E X, our semantic-awan attribute transfer model can be formally defined
G(M(2,Treff)) t)).
3.3 DIRECTIONAL. LATENT MAPPING
Only focusing on matching Ò with may cause irrelevant attribute changed. Therefore, we enforce the mapping network focus the change direction of semantic features in CLIP-space, and illustrate this idea in Figure T The details of the proposed directional latent mapping network is described as follows:
Architecture. It has been shown that the different layers of synthesis network, which correspends to different parts of the latent code, control different granularity of semantic feature. In order to better exploit this property. we design the directional latent mapping network, and depict it in Figure 2 To great extent, the degree of attributes disentanglement depends on the disentanglement of latent features. Therefore, we split the layers into groups, instead three (coarse, medium, and fine), with g fully connected mapping networks, one for each group. The divided part the latent code can be denoted as w [w',, W2: ..., Wg). so the mapping network is defined by:
(2) where M; is the i-th mapping network, and [+]] means concat operation. Then we use skip- connection operation to obtain the final manipulated latent code v = w + aM((w). and feed it the pre-trained StyleGANv2 generator get the final manipulated facial image á = G(t))
((w) = [Mr (w1), M2(w2), My..., (wg)))
Training Objective. The edited desired attribute is determined by the textual prompt t. Without paired training data, the mapping network could not correctly manipulate the latent code. In order to obtain extra powerful supervision, we use CLIP model to effectively extract semantic features (attributes) y of the corresponding images and text:
ys 91
Er(x;
(3)
Yt (t))),
4

---

Under review as a conference paper at ICLR 2022
CLIP-space
Reference
V
BBlakk hair"
Target
"Smile"
1
1
Result with "smile"
Result
without "black hair"
Figure 3: The illustration of attribute transfer loss LAT in our Semantic Directional Decomposition Network (SDD-Net). We embed both the text prompt and reference's target face into CLIP-spacc. When the projected vector Vy has the same direction with desired direction, the same attribute (e.g.. smile) is transferred to the target face. Reversely, the opposite attribute (e.g., black hair) will be transferred when the direction is inverse.
where denotes the pre-trained multi-modal feature extractor integrated in CLIP, y; and it denotes the extracted semantic features of and å, respectively. yt is the desired semantic feature extracted from textual prompt L.
In the latent CLIP-space, simply optimizing the matching degree between D: and ye may cause irrelevant attribute to be changed. Therefore, rather than optimizing the matching degree to the utmost extent, enforce the change direction between y and 0 to align with the yts and propose the semantic direction consistency (SDC) loss. The SDC loss is given by:
1/1 Vir
=
(4)
LSDC S(T,T),
where S(.,) is the similarity measurement. In this paper, use the effective cosine similarity as the measurement.
The above objective can guide the mapping network manipulate the latent code along the direction of textual prompt. In order to preserve the irrelevant parts, we use the following identity loss:
LID = (R(G(w)), R(G(w)))
(5)
where is pre-trained ArcFace DDng et al. 2019) network for face recognition, and (.:) com- putes the cosine similarity between its two arguments. Meanwhile, we use T.2 distance in W+ space control the degree of manipulation. Then, the whole training objective for directional latent mapping network is denoted
arg min Asnc t) Arall + (nD
(6)
wEW+
3.4 SEMANTIC DIRECTIONAL DECOMPOSITION
Inspired by the above method, we found that semantic attribute translation can be represented as a directional vector CLIP-space. Based on this, we leverage the same architecture directional mapping network, and propose facial attribute transfer method via semantic-aware knowledge transfer, named as semantic directional decomposition network (SDD-Net).
Given the textual prompt want to transfer the attribute, extracted from the reference image Øre/s to the target image Star- The Yref and Yaar are their corresponding semantic features respectively. And the semantic feature of textual prompt is y/-

---

Under review as a conference paper at ICLR 2022
"smile
Target
L2M-GAN
StyleCLIP=
Ours
Figure 4: Qualitative results for facial attribute editing on attribute "smile". The left column are the target images sampled from CelebA-HQ dataset. The other column from left to right are the editing results of L2M-GAN (Yang al. 2021a), StyleCLIPe (Patashnik et al. 2021), and our directional latent mapping network.
Due to the above direction- based manipulation, the attribute semantic feature projection problem. In the CLIP-space, a facial image can be treated as the composition of different semantic features. Therefore, assume that different facial images can be converted to each other in semantic-level. In other words, one semantic feature can be translated another semantic feature.
treat the facial attribute transfer problem as
Under such an assumption, we first define the RT (reference-target vector as: VRT = Dref Star- which is the variance of two semantic features, and also is the direction of translation. We the y as the projection direction. In order to extract knowledge from the reference, we project RT vector Ver onto desired direction Vi yr:
V [j
V
VIT
where V is the projected vector. Then we add it to the target semantic feature ytar to get the final goal:
Igoot = Ytar + 3V
(8)
where 3 is the hyperparameter.
Training Objective. We use same architecture of directional latent mapping network to obtain the manipulated image à G(M())), then match semantic feature ; = Ey with Ygont in CLIP-space. As illustrated in Figure the attribute transfer loss LAT is given by:
LAT MSE(), Vgont);
where MSE(.,)) -) is the mean squared error.
Meanwhile, we use the same 7.2 loss and identity loss to preserve the irrelevant parts. The whole training objective of SDD-Net is:
arg min AATLAT + dLLL2 + AIDLID- wEW+
(10)

---

Under review as a conference paper at ICLR 2022
Target
Result
Reference
"smile"
Target
Referenec
Result
Figure 5: The qualitative results of our SDD-Net single attribute transfer. Reference images in the left column. Target images the first row each parts, the rest is our manipulated images. Given the text prompt, our SDD-Net transfer the specific attribute ("smile"" the target images (top part) when the reference have the specific attribute. By contrast, the reverse attribute ("unsmiling") will be transferred (bottom part) when the specific attribute is not contained in the reference. Notice that, we only input one text prompt. So the SDD-Net could determine the forward or reverse transferring direction, according to the reference.
EXPERIMENTS
In this section, we first introduce the involved dataset and the implementation details. Then we will present the comparison results with several state-of-the-ard facial attribute editing and facial attribute transfer methods to prove the effectiveness of our proposed method. Finally, the ablation studies will be presented prove the effectiveness of method.
4.1 DATASET
In order to achieve text-driven facial attribute editing and transfer, we choose the widely-used CelebA-HQ (Krrras et al. 2017) dataset, which consists of 30,000 high quality facial images picked from the original CelebA (Liu al. 2013) dataset. The size of each high quality image is x024x 1024. In the original dataset, each image has 40 attributes annotations inherited from the original CelebA. However this work, remove these annotations, and leverage CLIP model as powerful supervision. We also use the standard training, validation and test splits inherited from CelebA dataset.
4.2 IMPLEMENTATION DETAILS
In this subsection we provide the implementation details of our proposed networks. All images taken from the CelebA-HQ are inverted by e4e (Tov et al., 2021). We set the batch size and the number of total iterations to and 50k respectively, during training. Our image editing is performed on StyleGANv2 pre-trained FFHQ (Karras et al., 2019) dataset. We keep the StyleGANv2 generator fixed during training. The CLIP model is pre- trained on 400 million image-text pairs. We use the Vision Transformer (VIT) (Dosovitskiy et al. 2020) and normal Transformer (Vaswani et al. 2017). which are integrated in CLP model. as our image encoder and text encoder, respectively. Our directional latent mapping module initialized using the pre-trained StyleGANv2, and trained

---

Under review as a conference paper at ICLR 2022
Reference
Target
StarGANv2
SDD-Net
Figure 6: Experiments on petrrence-guider image synthesis on CelebA-HQ. Reference and target images both in the first row. The second and third raw the synthetic results of StarGANv2 (Choi et al. 2020) and our SDD-Net respectively.
using Adam with the learning rate 5e-3. We set y 9, a = 0.1, and 8 = 130. For facial attribute
editing, the hyperparameters empirically set ASDC 1, AL2 0.4, and AID 0.02. For facial attribute transfer, the hyperparameters are empirically set as ÅAT = 1. AL2 = 0.2, and ÅιD = 0.02. Our methods trained on PyTorch with single TITAN RTX GPU. We optimize training objective through gradient descent, by back-proy pock-propggating the gradient through the pre-trained and fixed StyleGAN generator G' and CLIP multi-modal encoder E.
4.3 BASELINE METIJODS
Facial Attribute Editing Methods. We first compare our directional latent mapping network with the state-of-the-art facial attribute editing methods (i.e., 1.2MGGAN (Yang et al. (021a), and Style- CLIP (Patashnik et al.) 202T)) for the specific attribute: "smile". Due to the requirements of high- level semantic-aware knowledge, the smile attribute has become one of the most challenging global attributes. For L2M-GAN, we set the attribute domains as two ("smile" and "sad"), leveraging the domain label guide the manipulation. For StyleCLIP, we use the latent mapper network as the baseline model, which is marked as StyleCLIP*.
Facial Attribute Transfer Methods. The StarGANv2 (Choi al. 2020) learns to transform source image reflecting the style of a given reference image. We compare our SDD-Net with Star- GANv2 method on reference-guided image synthesis, which is challenging facial attribute transfer task. Reference-guided image synthesis requires that various high-level attributes in the reference images, such as hairstyle, makeup, beard and expression, should be transferrd to the target images, while the irrelevant information such as pose and identity should be preserved. For fair comparison, we resize the images to 256 reference-guided image synthesis.
4.4 RESULTS AND ANALYSIS
After careful compari- son, we have following observations: (1) The L2M-GAN performs well at disentangling attributes during editing. However, due to the strong constrain of orthogonal loss, the manipulated attribute not obvious. (2) Although the StyleCLIP* method can make the best use of semantic knowl- edge of CLIP-space, the irrelevant attributes are changed in facial attribute editing without correct guidance direction. (3) Our directional latent mapping network manipulates the attribute correctly and naturally, which demonstrates that the proposed semantic directional consistency (SDC) loss could enforce the editing model to change specific attributes while preserving irrelevant parts. also proves that there exists latent directions corresponding different semantic properties in latent space. By manipulating latent code along such direction or its opposite direction, we can add or remove attributes.
Facial Attribute Editing. The qualitative results are shown in Figure 4
8

---

Under review as a conference paper at ICLR 2022
Facial Attribute Transfer. Inspired by the above firection-based latent space manipulation for facial attribute transfer, we leverage this peculiarity for facial attribute transfer task.
We first execute our SDD-Net the "smile" attribute for single attribute transfer, and input the cor- responding text prompt to the model. The qualitative results are shown in Figure []] We observe that when reference has the consistent attribute appointed by the prompt, the SDD-Net could correctly transfer the specific attribute to the target face with irrelevant information preserved, as shown in the top part. On the contrary, when the reference has the opposite attribute, our SDD-Net also could transfer opposite attribute to the target without extra guidance. By fully leveraging the knowledge of CLIP-space, our SDD-Net could find the semantic-awe aware latent direction the latent space. The experiments show the excellent performance of our SDD-Net.
For the reference-guide guide image synthesis, we set the projected vector VIP is equal to the RT vector VRT. Figure [ provides qualitative comparison of the results. We observe that StarGANv2 mothed synthesizes images with same style code. However, the results show that StarGANv2 model ma- nipulates images limited space. As result, the attributes of the synthetic faces tend be exactly alike. In addition, the expression attribute such as "smile" is ignored during transferring. Compared StarGANv2, our SDD-Net transfers the multiple meaningful attributes each of the target faces in semantic-level. Meanwhile, our method manipulates the image along the latent direction lever- aging the knowledge CLIP latent space, which allows our method to synthesize realistic facial images, rather than simple style transfer.
4.5 ABLATION STUDY
We conduct qualitative ablation study, as shown in Figure 7 and show the significance of identity loss. We observe that, when hyperparameter ÅιD = 0.1. the ID loss hinders the attribute transfer. Then we experiment with Ån the attribute could be correctly transferred, but the identity information is changed. To trade-off, we set the ÅΙD 0.02.
"smile"
Reference
Input
(0.1)
(0.04)
Figure 7: The ablation study of identity loss. Under each column we specify (AιD) identity loss. Obviously the ID loss significant for facial attribute transfer.
CONCLUSIONS
In this paper, we first propose directional latent mapping network for text-driven facial attribute editing. By leveraging semantic direction consistency (SDC) loss, the directional latent mapping network could correctly edit relevant attribute while preserving irrelevant attributes. And on this basis, we propose semantic directional decomposition network (SDD-Net) for text-driven facial attribute transfer, which correctly transfers the semantic-aware attributes of reference image to the target image. Experiments show that our method achieves impressive performance on CelebA-HQ dataset.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432-4441, 2019.
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun- Zhu, and An- tonio Torralba. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727, 2020.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha. Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for nulti-domain image-to-image translation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8789-8797, 2018.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wor Ha. Stargan v2: for multiple domains. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8188 8197, 2020.
Diverse image synthesis
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recogmition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4690-4699, 2019.
Hui Ding, Kumar Sricharan. and Rama Chellappa. Exprgan: Facial expression editing with con- trollable expression intensity. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Garoe Dorta, Sara Vicente, Neill DF Camphell, and Ivor JA Simpson. The gan that warped: Seman- tic attribute editing with unpaired data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5356-5365, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.119999 2020.
Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defi- nitions of cognitive image properties. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5744-5753, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: interpretable gan controls. arXiv preprint 000:20004.02446, 2020.
Discovering
Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen. Attgan: Facial attribute editing by only changing what you want. IEEE transactions on image processing, 28(1 (1 1):5464- 2019.
Zhenliang He, Meina Kan, Jichao Zhang, and Shiguang Shan. Pa-gan: Progressive attention gener- ative adversarial network for facial attribute editing, arXiv preprint arXiv:2007.05892, 2020.
Bingwen Hu, Zhedong Zheng, Ping Liu, Wankou Yang, and Mingwu Ren. Unsupervised eyeglasses removal in the wild. IEEE Transactions Cybernetics, 2020.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" "sterrabiitt"" of generative adversarial networks. arXiv preprint 197700017, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv:1710.10100 2017.
10

---

Under review as a conference paper at ICLR 2022
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 8110-8119, 2020.
Jeong-gi Kwak, David Han, and Hanseok Ko. Cafe-gan: Arbitrary face attribute editing with complementary attention feature. In Computer Vision-ECCV 2020: 16th European Conference, Springer, 2020.
Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp-
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey. St-gan: Spatial transformer generative adversarial networks for image compositing. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9455-9455555 9464, 2018.
Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, and Shilei Wen. Stgan: A unified selective transfer network for arbitrary image attribute editing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3673-3682, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Deep learning face autributes in the wild.
Yotam Nitzan, Rinon Gal, Ofir Brenner, and Daniel Cohen-Or. Large: Latent-based regression through gan semantics. arXiv preprint arXiv:2107. 1186, 2021.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. arXiv preprint arXiv:2103.1 10221003.99 77249, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. arXiv preprint (xXix:2103000022 2021.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2287-2296, 2021.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 9243-9252, 2020.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1--4 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information 2017.
processing systems, pp.
Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12863-12872, 2021.
Taihong Xiao, Jiapeng Hong, and Jinwen Ma. Elegant: Exchanging latent encodings with gan for transferring multiple face attributes. In Proceedings of the European conference on computer vision (ECCV), pp. 168-184, 2018
Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou. Generative hierarchical features from synthesizing images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4432 4442, 2021.
11

---

Under review as a conference paper at ICLR 2022
Guoxing Yang, Nanyi Fei, Mingyu Ding, Guangzhen Liu, Zhiwu Lu, and Tao Xiang. L2m-gan: Learning to manipulate latent space semantics for facial attribute editing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2951-291299 2021a.
Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 672-681, 2021b.
Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing. Mark Hasegawa-Johnson, and Minh N Do. Semantic image inpainting with deep generative models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5485-5493, 2017.
Weidong Yin, Ziwei Liu, and Chen Change Loy. Instance-level facial attributes transfer with geometry-aware flow. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 33, pp. 9111-9118, 2019.
12


Reviews:
[{"review_id": "x8-PG2j3XS4", "reviewer": "Reviewer_V8mm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-- the proposed semantic directional decomposition strategy seems interesting and novel. \n\n-- the experimental results reported in this submission seem better than prior works such as StyleCLIP.\n\nWeakness:\n-- more experiments with various attribute transfer are expected to further demonstrate the efficacy of this proposed method. In this submission, only a few attributes are considered (\"black hair\", \"bang\", \"smile\"). \n\n-- it seems that this proposed work can only handle one single attribute change/transfer? can it work when multiple attributes need to be transferred?", "summary_of_the_paper": "This submission proposes a novel test-drivern directional latent mapping network with a new semantic direction consistency constraint for semantic-level facial attribute transfer. By treating the facial attribute transfer problem as an attribute semantic feature projection problem, a semantic directional decomposition strategy is proposed.", "summary_of_the_review": "This submission proposes a novel test-drivern directional latent mapping network with a new semantic direction consistency constraint for semantic-level facial attribute transfer. By treating the facial attribute transfer problem as an attribute semantic feature projection problem, a semantic directional decomposition strategy is proposed. The reported experimental results seem good, especially comparing to the SOTA like StyleCLIP.\n\nI have some concerns about the experiment settings and I feel some more discussions are needed. On one hand, only a few attributes are considered in the current submission, more experiments with various attribute transfer are expected; on the other hand, it is expected to know whether this method can still work when more than one attributes need to be transferred.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "j3cl3fDM50j", "reviewer": "Reviewer_ksFh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe idea and method are intuitive and easy to follow. Figures are pretty easy to understand.\nThe selected results are impressive.\n\nWeaknesses:\nLittle novelty and confusing statements. The main contribution is essentially a loss term that measures latent similarity/distance in CLIP latent space. The other parts are from existing works, including StyleGANv2, e4e, and CLIP. Moreover, it seems unreasonable that text feature y_t can be treated as a semantic direction. Besides, the role of the reference image in attribute transfer is confusing since the text prompt plays a significant role in the presented results.\nInsufficient evalutions. The authors have not justified whether their method works on a wide range of attributes with quantitative evaluations.\nHard to use in practice. Based on my understanding, the model needs re-training for different attributes and different controlling degrees, making it hard to use in practice.", "summary_of_the_paper": "This paper proposes a new loss function for unsupervised facial attribute editing and transfer. Specifically, a latent mapping network is trained by optimizing the similarity/distance between the generated and desired image in the CLIP latent space. Experiments show some comparisons and ablations for the \"smile\" attribute.", "summary_of_the_review": "I think this paper has limited novelty and insufficient evaluations. The detailed comments are listed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "WwR5DNiPVnX", "reviewer": "Reviewer_nZQ1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nstrengths:\n1. The incorporation of attribute-aware mechanism based on StyleCLIP is technically sound.\n2. Qualitative results are provided, demonstrating the proposed method performs fine on CelebA-HQ dataset.\n\nweaknesses:\n1. The comparison to StyleCLIP is not sufficient, especially considering this paper is a follow-up work of StyleCLIP and there are no quantitative experiments to support the authors' claims. For instance, only 4 examples in Figure 4 demonstrate the compairson to StyleCLIP. From my view, the outputs of StyleCLIP and the proposed method don't have a big difference. If the authors would like to claim that their method can edit relevance and keep irrelevance while StyleCLIP cannot, it would be better to point out in the Figure which irrelevant parts of faces are changed by StyleCLIP. I would recommend the authors include at least 20 images in the supplementary to demonstrate the effectiveness of their approach. (I didn't find the supplementary materials in the current version)\n2. The paper only conducts experiments on CelebA-HQ dataset and the title is \"text-driven image manipulation\" rather than \"text-driven celebrity face manipulation. I would recommend the authors perform experiments on more datasets such as cars dataset [1] or dogs dataset [2] following StyleCLIP.\n3. Typos like \"definded\" in the paper should be corrected. Besides, math symbols should be explained in advance. For example, the mapping network \"M\" is first mentioned in Section 3.2 and the authors didn't point out the definition of \"M\" until Section 3.3.\n\n\n[1]Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015\n[2] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. In Proc. CVPR, pages 8188–8197, 2020.\n\n", "summary_of_the_paper": "This paper proposes a latent mapping mechanism based on StyleCLIP to disentangle the semantic attributes of human face. The authors provide some qualitative results and ablation study on CelebA-HQ dataset.", "summary_of_the_review": "Overall I tend to reject the paper. Please refer to the Main Review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9dIyi0joEkz", "reviewer": "Reviewer_wMtJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Personally, the paper was enjoyable. It was intuitive and a clever idea, end-to-end. The figures depicting the methods and theory are informative; the experiments seem convincing, and the overall framework seems reproducible.\n", "summary_of_the_paper": "The authors proposed a directional latent mapping network for facial attribute editing via text inputs. The directional latent mapping network could correctly edit relevant attributes while preserving irrelevant attributes via training with the semantic direction consistency (SDC) loss. This paved the way to a novel semantic directional decomposition network (SDD-Net) for text-driven facial attribute transfer: SDD-Net transfers semantic-aware attributes from reference images to a target, with the multi-modal approach guiding the process via text input descriptions. CelebA-HQ dataset was used to compare results with recent SOA methods.", "summary_of_the_review": "\nAfter reading the paper for the first time, I figured a sure \"accept.\" However, I then noticed they always used it for \"smile,\" besides when \"not smiles\" (i.e., any attribute but \"smiles\" were transferred. Not that this should take everything away from the work. Nonetheless, I believe the authors are not transparent enough in this aspect of the paper-- do we need K models for K attributes? The authors attempted to show \"black hair\" and \"bangs\" but it was unclear what was important in the results.\n\nThey showed many image montages throughout: it is good to highlight the 'take-home message.' Too many times was I zooming in trying to understand the differences between samples. A better explanation of the variations is needed.\n\nI cannot help but see this work as extremely high in potential: a potential that far surpasses the quality of the current version. More insight, edge cases, and failure modes would be a plus. More attribute types would be best. Which attributes work, which doesn't, and how does this relate to the motivation (i.e., global vs. local). \n\nIf this was FG I would suggest \"accept,\" but as the paper shows plenty, but does not explain enough. It's only being used on smiles for most of the time, with no emphasis on this, just seemed suspect (not that it was intentional, but just was a letdown when I finished the first pass, was excited about the paper, and then realized this which was a total let down)..", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a new loss function for facial attribute GAN editing and transfer via text inputs. 
A latent mapping mechanism based on StyleCLIP is used to disentangle the semantic attributes of human face. 
The resulting semantic directional decomposition network (SDD-Net) transfers attributes from reference image to a target guided with text descriptions. Experiments show on CelebA-HQ dataset some qualitative results and ablations  for the « smile » attribute.

The main contribution is essentially a loss term that measures latent similarity in CLIP latent space.
Most of the reviewers are not convinced by the approach and have raised several issues. One can question the relevancy of the way that text features are used (as a semantic direction). The role of the reference image in attribute transfer is is also questionable in the proposed framework.
Additionally, evaluation is not sufficient, in particular to investigate whether the proposed method works on a wide range of attributes.
The paper only conducts experiments on CelebA-HQ dataset. It would be interesting to have experiments on other  datasets.
The comparison to StyleCLIP is also insufficient, and there are no quantitative experiments to support the authors' claims. 
We encourage the authors to take into account all these remarks and Rs' comments in order to get an improved proposition for a future conference.
Decision: Reject