One Model to Edit Them All: Free-Form Text-Driven Image Manipulation with Semantic Modulations
Authors: Yiming Zhu
Hongyu Liu
Yibing Song
Ziyang Yuan
Xintong Han
Chun Yuan
Qifeng Chen
Jue Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Free-form text prompts allow users to describe their intentions during image manipulation conveniently. Based on the visual latent space of StyleGAN[21] and text embedding space of CLIP[34], studies focus on how to map these two latent spaces for text-driven attribute manipulations. Currently, the latent mapping between these two spaces is empirically designed and confines that each manipulation model can only handle one fixed text prompt. In this paper, we propose a method named Free-Form CLIP (FFCLIP), aiming to  establish an automatic latent mapping so that one manipulation model handles free-form text prompts. Our FFCLIP has a cross-modality semantic modulation module containing semantic alignment and injection. The semantic alignment performs the automatic latent mapping via linear transformations with a cross attention mechanism. After alignment, we inject semantics from text prompt embeddings to the StyleGAN latent space. For one type of image (e.g., `human portrait'), one FFCLIP model can be learned to handle free-form text prompts. Meanwhile, we observe that although each training text prompt only contains a single semantic meaning, FFCLIP can leverage text prompts with multiple semantic meanings for image manipulation. In the experiments, we evaluate FFCLIP on three types of images (i.e., `human portraits', `cars', and `churches'). Both visual and numerical results show that FFCLIP effectively produces semantically accurate and visually realistic images. Project page:  https://github.com/KumapowerLIU/FFCLIP.

One Model to Edit Them All: Free-Form Text-Driven Image Manipulation with Semantic Modulations
Yiming Zhu
Hongyu Liu
Yibing Song
Ziyang Yuan
Xintong Han
Chun Yuan
Qifeng Chen? Jue Wang?
Tsinghua Shenzhen International Graduate School
Hong Kong University of Science and Technology
Tencent Lab *Huya Inc
zym2 0 @mai 1s tsinghua........ edu.cr cn bliudgecse. ust.hk yibingsong. evêema com yuancusz twinghuu..u........ edu. en.
Drard blodd mobaws bair
Inpux
up
Angry
Inpur
From Stxtes
Sports
White leep
leep
From Sixties
Impur
Byoomine
Romanoqpe:
Gotlii
Banquu
Choars
Snow
Sharp Spines
Orim-sappe deme
The proposed FFCLIP model edits each type of image with free-form text prompts. For each input image shown on the first column, we show manipulation results with text prompts on their corresponding row. Each text prompt, which contains single semantic meaning (e.g., 'Blue') during training, can convey multiple semantics (e.g., 'Black from sixties' and 'Beard bond mohawk hair") during inference for free-form image manipulations.
Figure 1:
Abstract
Free-form prompts allow users to describe their intentions during image ma- nipulation conveniently. Based on the visual latent space of StyleGAN [21] and text embedding space of CLIP [34], studies focus how map these la- tent spaces for text- rext-driven attribute manipulations. Currently, the latent mapping between these spaces is empirically designed and confines that each manipu- lation model can only handle one fixed text prompt. In this paper, we propose a method named Free-Form CLIP (FFCLIP), aiming establish an automatic latent mapping so that one manipulation model handles free-form text prompts. Our FFCLIP has cross- pross-todality semantic modulation module containing semantic alignment and injection. The semantic alignment performs the automatic latent mapping via linear transformations with a cross attention mechanism. After align-
'Y. Zhu and H. Liu contributes equally. 'Y. Song and C. Yuan are corresponding authors. This work is done
when Y. Zhu is an intern in Tencent Al Lab.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ment, we inject semantics from text prompt embeddings to the StyleGAN latent space. For one type of image (e.g.. 'human portrait',, one FFCLIP model can be learned to handle free-form text prompts. Meanwhile, we observe that although each training text prompt only contains a single semantic meaning, FFCLIP can leverage text prompts with multiple semantic meanings for image manipulation. In the experiments, we evaluate FFCLIP on three types of images (i.e., 'human portraits', "cars', and 'churches'). ). Both visual and numerical results show that FFCLIP effectively produces semantically accurate and visually realistic images. Project page: https:/ / /gi thub. com/ Kumapowe LIU/ /FFCU/FF.PPPP.
1 Introduction
Neural image synthesis has received tremendous investigations since the Generative Adversarial Networks (GANs) [14]. The synthesized image quality is significantly improved via the StyleGAN- based approaches [22, 21, 203. Recently, free-form text prompts describing user intentions have been utilized to edit StyleGAN latent space for image attribute manipulations [33, 533. With a single word (e.g., 'Blue') or phrase (e.g.. 'Man aged 10') as an input, these methods edit the described image attribute accordingly by modulating the latent code in StyleGAN latent space.
The accurate image attribute manipulation relies on the precise latent mapping between the visual latent space of StyleGAN and the text embedding space of CLIP. An example is when the text prompt is 'Surprise,' we first identify its related semantic representations (i.e., 'Expression") in the latent visual subspace. Then, we modulate the latent code of this identified latent subspace via the text embedding guidance. Pioneering studies like TediGAN [53] and StyleCLIP [33] empirically identify which latent visual subspace corresponds to the target text prompt embedding (i.e., attribute-specific selection in TediGAN and group assignment in StyleCLIP). This empirical identification confines that given one text prompt. they must train a corresponding manipulation model. Different text prompts require different manipulation models modulate the latent code in latent visual subspace of StyleGAN, Although the global direction method in StyleCLIP does not employ such a process, the parameter adjustment and edit direction are manually predefined. To this end, we are motivated to explore how to map text prompt embeddings to latent visual subspace automatically. So a single manipulation model is able tackle different semantic text prompts.
In this paper, we propose a free-form method (FFCLIP) that manipulates one image according to different semantic text prompts. FFCLIP consists of several semantic modulation blocks that take the latent code w in StyleGAN latent space W+ [1] and the text embedding as inputs. Each block has one semantic alignment module and one semantic injection module. The semantic alignment module regards the text embedding as the query, the latent code w as the key, and the value. Then we compute the cross attention separately in both position and channel dimensions to formulate two attention maps. We use linear transformations to perform a latent mapping from text prompt embedding and latent visual subspace, where the linear transformation parameters (i.e., translation and scaling parameters) are computed based on these two attention maps. Through this alignment, we identify each text prompt embedding to its corresponding StyleGAN latent subspace. Finally, the semantic injection module modifies the latent code in subspace via another linear transformation following HairCLIP [51]. The modulated semantics are represented as the latent code offset Aw, which is refined progressively through several semantic modulation blocks.
From the perspective of FFCLIP, the empirical group selection of w in [33, 51] is a particular form of our linear transformations in the semantic alignment module. Their group selection operations resemble a binary value of our scale parameters to indicate the usage of each position dimension of w. On the other hand, we observe that the W+ is not disentangled completely, the empirical design could not find the precise mapping between StyleGAN's s latent space and CLIP's text semantic space. In contrast, the scale parameters in our semantic alignment module adaptively modify the latent code w to map different text prompt embeddings. The alignment further improved via our translation parameters. We evaluate our method on the benchmark datasets and compare FFCLIP to state-of-the-are methods. The results indicate that FFCLIP superior in generating visually pleasant content while conveying user intentions.
2

---

2
Related Works
Latent Space Image Manipulation. There are a wide range of image manipulation and restoration studies in the literature [29, 31, 30] and we focus on discuss the StyleGAN based methods here. The latent space in StyleGAN [22, 21, 20, 26] has demonstrated great potential in representing the semantics in the image, motivating many works to disentangle the latent space for controllable image manipulation. Specifically, investigations [40, 18, 3, 45, 4] predict the meaningful offsets or directions in the latent space given image annotations as supervision, while studies [41, 48, 16, 49] disentangle the latent space in an unsupervised manner to find the semantic directions. Although these methods achieve great performance in latent space manipulation, they can only find limited semantic directions that confine user intentions. contrast, we achieve free-form image manipulation conditioned on arbitrary text prompts, giving users more degrees of editing freedom. Our StyleGAN inversion encoder is related to GAN inversion methods [1. 2, 5, 6, 38, 46, 50] that maps the image to the latent code space E W [1]. In our work, we use the e4e [46] as StyleGAN inversion encoder following StyleCLIP [33], and introduce the cross attention mechanism [47, 13, 28] to linearly map the text and visual latent space for free-form editing.
Text-driven Image Generations. Starting from [37] that leverages text embeddings as the condition for GAN- based image training, several works [56. 57, 58] improve the synthesis quality by introducing multi-scale or hierarchically-neste
GANs. And attention modules are introduced in [54] to match the generated images to text. [8. 32, 27, 44] improves image content fidelity considered from the network structure and training perspectives. The text to image generation performance is further boosted in DALLE [36, 35] and DiffusionCLIP [23] where text prompts with multiple semantic guidance are translated into images. Vision transformer [10] and diffusion model [15] also show impressive results in text-driven image manipulations. TediGAN [53] transfers the image and text to a shared StyleGAN latent space, and modulates the image latent vector with text. Recently, CLIP [34] constructs text-image embedding space to connect the semantics between image and text embeddings. Combining the CLIP model with StyleGAN becomes promising for text-driven image manipulations. Pioneering works such as StyleCLIP [33] use the CLIP model to map semantics from text to images. HairCLIP [51] builds top of StyleCLIP while focusing more on the hair regions with reference images and semantic injections. FEAT [17] proposes an attention mask to prevent changes unedited areas. These methods need manually select specific StyleGAN latent subspace according to the target text. So they fail to train a single model for different text prompts. In comparison, FFCLIP can adaptively match one text embedding to the desired latent subspace, making one model process multiple text prompts for the corresponding image manipulations.
3 Proposed Method
Fig. 2 shows an overview of the proposed method. Given input image and text prompt, we can obtain the StyleGAN latent code w and text embedding e,. We propose k (we set k = 4 in practice) semantic modulation blocks where each block first aligns semantics between w and e; and then edits the latent code. The output of each block is an offset Aw which adds to the latent code w. Finally, a StyleGAN generator decodes the resulting latent code the manipulated image. In the following, we first briefly review the StyleGAN and the CLIP model, then we illustrate how we align and inject semantics for latent space editing.
StyleGAN Revisiting,
Studies for StyleGAN [22, 21, 20] generate visually realistic face images with various semantics. In StyleGAN, a latent variable Z E g-512 from the Gaussian distribution is gradually transformed to disentangled latent code W E 1252 with semantic meanings. The latent code W is then fed into 18 layers for image generation. Later, investigations [46] find that an extension of W (i.e., W+ E R 18x512) is more effective for identity preservation when manipulating the image. There is a definite relation between the layers of W+ and the semantic latent subspace. So the semantic modulation can be formulated editing some specific layers of W+ As a result, image manipulation methods [3, 45, 4, 18] based on StyleGAN invert an input image to a latent code w E W for semantic modification. We follow [33] utilize the StyleGAN2 our generator.
CLIP Revisiting, The CLIP model [34] is pretrained from 0.4 billion image-text pairs for cross- modality semantic matching. This model consists of an image encoder and a text encoder to project an image and a text prompt into 5 512-dimension embedding, respectively. We leverage the text
3

---

CLIP Tast Enooder
"Real hoir" Text Prompt
Ce
AW
StyleGAN Inversion Encoder
Semantic Aligmment
Semantic
StyleGAN
Injection
Gecerator
Inpot Image
Semantic Modulation Block
Manipulated Image
Figure 2: Overview of our pipeline. Given a text embedding e, and an input latent code w, propose several semantic modulation blocks to produce an offset Aw for latent space modification. In each block, there is semantic alignment module automatically map the semantic subspace in based on Then, the semantic injection module modifies this latent subspace to refine Aw.
encoder of CLIP model to produce the text prompt embedding. This encoder is versatile in perceiving text prompts with different semantic meanings.
3.1 Semantic Modulation
Fig. shows the framework of semantic modulation block where there are k blocks in total. The inputs are an image latent code w E W+ converted from image with e4e [46], and text embedding e, extracted from the CLIP model. use semantic modulation blocks progressively refine Aw. The Aw will modify for latent subspace editing. The output of the i-th block is denoted as Aw;. In each block, there is a semantic alignment module followed by a semantic injection module, which are illustrated as follows:
Semantic Alignment Module. The automatic semantic correspondence between 6, and enables that one model is able to process text prompts with different semantic meanings, We establish this automatic correspondence by aligning w and e, via a linear transformation. The parameters of this linear transformation are learned based on the cross attention measurement. Specifically, for the semantic alignment module in the i-th block, we compute a cross attention map between and Aw;-1 (Awy = w) in both position and channel dimensions similar to the dual attention network [11]. The scale and translation parameters of the linear transformation are computed based on the cross attention maps.
For the position dimension, set the as the Query Qn R512 and the latent code Aw;-1 the Value V E 188x512 and the Key K E 18x512 Then, we compute the cross attention map Attention, as scale parameter 5 E 118, The scale parameter S models the Value V in position dimension for transforming the latent space to match the text embedding semantics. The cross attention computation on the position dimension can be written
Qp = R K = Aw; w . Aw;-WW
Attention, Softmax(Q (T,
S Attentionp:
where Wº,, W WV E R512x512 We use these scale parameters to adjust the contribution of each dimension of W+ conditioned on e,. Meanwhile, observe that the W space is not fully disentangled. One semantic property is reflected not only specific element of w, but other elements well. To this end, compute the translation parameters in the channel dimension to enhance the semantic alignment.
For the channel dimension, we use the same K and V as those in the position dimension, and new Query Q. E 118 to calculate the cross attention map Attention, in the channel dimension. Then, reconstruct the Value V with the attention map Attention, and utilize adaptive average pooling

---

S
1x312
O
1x512
Mutai
Softhlas
18x1
18x
Loyer Nom
V
2W,1
18X 512
18X
18x512
V
10x512
1X512
Semantic Injection
18x512
9
512x
1X512
Semantic A lignment
Figure 3: Our semantic modulation block consists of semantic alignment module and semantic injection module. Given Aw;-1 and e; as input, we compute their cross attentions in position and channel dimensions to learn a linear transformation. This transformation aligns Aw;- to Then, we use semantic injection to refine Aw;- to Aw; for the input to the next block.
compute the translation parameters T E RR512 These processes can be written
Q. = e T K Aw; WK Aw;- WV
Attention, = Softmax(Q.K),
AAP(Attention,
where the W and WV the same the those in the position dimension, and the W R AAP the adaptive average pooling operation. After we get the scale and translation parameters, we can align Aw;-1 with a linear transformation as follows:
= S x V +T,
where the is the output of this semantic alignment module. Then, we inject to this aligned latent subspace via our semantic injection module in the following.
Semantic Injection Module. In this module, we inject the semantic from é, to F; following that in HairCLIP [51]. Specifically, adopt the fully-connected layers to map the text embedding e into two injection parameters 3 E R252 and E R8512, respectively. Then, we inject the text embedding
semantics follows:
Aw; (1 3t
Ma. B
dz.
where Ma, and da, represent the mean and the variance of the input T., respectively. With the aligned semantics, our injection produces the Aw; to modify StyleGAN latent code. The Aw; is further refined through the following semantic modulation blocks obtain Aw for the final image generation.
3.2 Training Objectives
In text-driven image manipulation, we focus aspects of the output result. First, the semanties of the target object in the image shall be consistent after manipulations. Second, the output image shall be semantically relevant the text prompt. To this end, we follow [51] and develop two types of loss functions, i.e., semantic preserving loss and text manipulation loss.
Semantic Preserving Loss. We aim to preserve the semantic consistency between the input and output images. This semantic shall represented within CLIP model. Specifically, we feed the input and output images through the CLIP image encoder to get two image embeddings. If the image
5

---

subject is consistent, the distance between these two embeddings should be small. So we reduce the distance between these two embeddings. We call it as image embedding loss and define as follows:
Lembd = 1 - cos E (G(w")),ET ECLIP E (G(w))}
(5)
where the G(-) denotes the pretrained StyleGAN generator, cos(.) means cosine similarity, F.LIP() is the pretrained CLIP image encoder, and w' + Av are the input and our edited StyleGAN latent codes, respectively. Then we utilize the L1 norm for preserving the irrelevant semantic regions:
Foorm |
Moreover, for 'human portrait' images, the background loss and the face identity loss are used to improve the performance, and the face identity loss is defined as follows:
La = 1 - cos where R()) indicates the pretrained ArcFace network [9]. Also, the background loss can formulated as:
c (R(G(w')) R(G(w))),
Cbg = ((G((") - G(w)) (P(G(w")) n P(G
(8)
where P(.) the facial parsing network [25] and P(G(w"), P(G(w)) represents the non-facial
regions in the input and output images, respectively. The overall semantic preserving loss Sap can be written as
Lup Aembd Tremb Anorm Tmnor Apa Ltd
Aag
Ligg
where dembd. Anorm Aed, and Abg the weights that adjust the contribution of each loss term. We set Aembd = 1.0, Anorm = 1.5, Aid = 1.0, and Aby = 2.0. particular, we only introduce Lid and Lbg when we edit face images.
Text Manipulation Loss. To evaluate the correlation between output images and text prompt embeddings, we minimize their cosine distance by using the CLIP model. can be written as:
L. = 1 cos (G(()(GGWW"))))))) eth.
(10)
where (.LIP() is the pretrained CLIP image encoder. Overall, the total loss is
Ltotal = Xap Lep + d Lt:
(11)
where Aep and d₂ are set as 1 and 1.5 in our method, respectively.
Experiments
We first illustrate implementation details. Then we compare FFCLIP with existing methods qualitatively and quantitatively. An ablation study validates the effectiveness of our modules. More results and video demo are provided in the appendix and supplementary files, respectively. We will release our implementations to the public.
4.1 Implementation Details
We train and evaluate FFCLIP on the CelebA-HQ dataset [19], the LSUN cars dataset [55], and the LSUN Church dataset [55]. To invert images into StyleGAN2's latent codes, we use pretrained e4e [46] as the image encoder. The dimensions of the latent code 18 x 512, 16 512 and 14 x 512 for face, car and church images, respectively. When training our model, we leverage the text encoder from CLIP, and use the pre-trained StyleGAN2 generate edited images. For each dataset, we use the corresponding pre-trained StyleGAN2 as the generator. We use 44 text prompts for face images containing emotion, hair color, hairstyle, age, gender, make-up, etc. Meanwhile, we follow the text prompts from StyleCLIP [33] when editing LSUN cars and church datasets. We randomly choose text prompt for an input image for model training. practice, we use a multi-step learning rate with an initial learning rate of 0.0005. The Adam [24] optimizer is utilized with 31 and ,22 set to 0.9 and 0.999, respectively. For CelebA-HQ dataset, the total training iterations are 150,000 and the batch size is 8. For LSUN church dataset, the total training iterations are 200,000 and the batch size is 4. For LSUN cars dataset, the total training iterations are 100,000 and the batch size is 8. We train our model on a workstation with 8 Nvidia Telsa V100 GPUs.

---

Ixput
FECUIP
StAcCLIP
TeIGAN
Tiput
RCCP
SUICUP
Augo
Agut
Dalee
Figure 4: Visual comparison with TediGAN [53] and StyleCLIP [33] on the CelebA-HH dataset. The text guidance is described the left side of each row. FFCLIP is more effective to produce semantic relevant and visually realistic results.
4.2 Qualitative Evaluation
We compare our method to state-of-the-art text-guided image manipulation methods TediGAN [53] and StyleCLIP [33]. Then, show results with diversified text prompts. We also show model is able to manipulate images with a single text prompt containing multiple semantic meanings.
State-of-the-an Comparisons. Fig. 4 shows the visual comparison results. We observe that the manipulated content does not exactly match the text prompt in the TediGAN's results. An example is shown on the first row with the 'Angry' prompt. The beard appears on the man in the TediGAN's result while it does not exists in the input image. While StyleCLIP improves the manipulation result of TediGAN, still limits editing input images with high semantic accuracy. The hair still presents in the StyleCLIP result with the 'Bald' prompt. In comparison, our FFLIP is effective in editing images based on the text prompt semantics while maintaining visual realism. Compared to existing methods of training each model for text prompt, our model processes multiple text prompts with only one model. Our superior performance results from the accurate semantic alignment across the text embedding and StyleGAN latent space.
Free-form Image Manipulations. Fig. 5 shows our image manipulation results where FFCLIP process text prompts with different semantic meaning. For human portrait images, FFCLIP preserves the face identity, produces high-quality edited images, and manipulates images with different seman- ties. Meanwhile, results transfer. The various manipulation results in multi datasets prove the robustness of our method. We pioneeringly develop a single model to process different text prompts for one type of images. More results are shown in Appendix.
and churches datasets realistic with the accurate semantic
Text Prompts with Multiple Semanties. We observe that although we train our model with single word text prompt, FFCLIP is able to edit images with text prompts containing multiple semantics. Fig. 6 shows the results. FFCLIP simultaneously transfers multiple text prompt semantics the input image with realistic appearance. This success is because of our accurate semantic alignment for the text embedding from the CLIP model. More results are shown Appendix.
Interpolation Results. FFCLIP can achieve fine-grained image manipulation by interpolating with two output latent codes. As shown in Fig. 7, generate the intermediate latent code by linear weighting w w + A(wí w)). where w/ is the intermediate output, w and w/, are the outputs of FFCLIP which correspond two different manipulation semantics. By gradually increasing the blending parameter X from 0 to 1, we can generate the results between two semantics (e.g., the visual results between 'Aged 10" and 'Aged 80'). More interpolation results shown in Appendix.
4.3 Quantitative Evaluation
There hardly a straight quantitative measurement to evaluate the image manipulation results. Nevertheless, in the text-driven image manipulation scenario, we believe the image results should
7

---

leput
Diseat
Aqud
Blood
Mehask
Baid
Cudy
Toslor
Ineatt
Byzarlize
Remenequu
Goflic
Banoue
Cleads
Suow
Sham Spine
Oraondiee coeas
Inpst
Sports
Figure 5: Our manipulation results CelebA-HQ. LSUN cars and churches datasets with different text prompts. The input images are shown the first column and our results are shown in the corresponding row. Our results are highly semantically relevant the text prompts while maintaining visual realism.
Inpot
FESLIP
Ipput
FFCLIP
Inpui
PRCLIP
Input
FRCLIP
the Roehheree
Symtto dick
Figure 6: Our manipulation results with text prompts containing multiple semantic meanings. Our model, trained with text prompts consisting of a single semantic, is able to manipulate images based on multiple semantic meanings.
correspond to the text prompt semantics. We utilize several text prompts and randomly select 1000 testing images from the CelebA-HQ dataset. For each prompt, we produce the results from TediGAN, StyleCLIP and ours. Then, we follow [42] to use the multiple semantic classification models to measure the text-relevance of these results.
Table shows these evaluation results under the editing performance column. We use different configurations to compare these results. When the text prompt 'Bald', we use the PSPNet [59] to locate the hair region and count the number of pixels within this region. When the text prompt is 'Red hair', we compute the average color difference of the hair region each result. When the text prompt is 'Angry". we use ESR [43] to determine whether the expression of the results belongs to the angry

---

Input
Aged
Aged
Input
Higgy
Angy
Figure 7: Interpolation results. We generate the intermediate results between 'Aged 80' and 'Aged 10' the first row, and the results between 'Happy' and 'Angry' the second row.
Table 1: Quantitative evaluations the CelebA-HQ dataset. FFCLIP is more effective to produce semantically relevant results for both editing performance and human subject evaluation. +.' denotes that quantitative comparison is given as corresponding classifiers available.
Editing Performance
Human Subject Evaluation
Text Prompt
TediGAN
StyleCLIP
Ours
TediGAN
StyleCLIP
Ours
Bald
0.2507
0.1015
0.0279
5.7%
8.6%
85.7%
Angry
0.4520
0.4860
0,5810
0.0%
17.1%
82.9%
Red hair
1,0971
1,0416
0.7171
2.9%
0.0%
97.1%
Aged 10
21.1448
17.0053
9.8874
2.9%
0.0%
97.1%
Bowl cut hairstyle
2.9%
5.7%
91.4%
Beard blond mohawk hair
8.6%
0.0%
91.4%
category. When the text prompt is 'Aged 10', we use an age classification method [39] to compute the distance between the estimated ages from the results and 10. For each prompt, compute the quantitative numbers of all the results, As shown in this table, our method is more effective to transfer text prompt semantics to the images.
Human Subjective Evaluation. Besides designing different metrics to evaluate semantic transfer, we conduct human subjective evaluations on the manipulated results from compared methods. We randomly collected 54 images which
manipulated based on 6 text prompts. 35 participants with diverse backgrounds are asked to vote for the best results based on the three equally important principles. First, they should select the result where the semantic meaning corresponds the text prompt most. Second, they should select the result where the human identity is best preserved. Third, they should select the most visually realistic image. We tally the votes as shown in Table I under the human subject evaluation column. It indicates that most participants favor our results compared to others, validating the effectiveness of our semantic modulation blocks. Moreover, we conduct other human subjective evaluations with more text prompts in Appendix.
4.4 Ablation Analysis
Effect of Semantic Alignment Module. The proposed semantic alignment module modifies Style- GAN's latent code w from the position and channel dimensions by the scale and translation parameters. In order to verify the necessity of modulation with scale and translation, we trained three models: without scale parameter, without translation parameter, and without both parameters, respectively. It is worth mentioning that all three models contain a semantic injection module to inject text information into the latent space. As shown in Fig. 8, without scale or translation parameter, we cannot find the latent subspace for the text accurately (e.g., in rowl col4, 'bald' latent subspace latent subspace we find is not disentangled (e.g., in rowl col3, the face ID changes; in rowl col8, the hair color changes; and the result in rowl co19 wears glasses, etc.). Furthermore, only a complete model with both scale and translation parameters can be competent given combination of texts unseen during training (e.g., beard blond mohawk hair in row2). The associated numerical results are shown in Table 2.
found), and the

---

Imput
we
S82T
Out
wbb
wo SEIT
Figure 8: The effect of semantic alignment module. Input texts given the left, "w/o S' means without scale parameter, 'w/o T means without translation parameter, and 'w/o S&T means without both. The results show that scale and translation modulation essential for latent mapping.
Irpee
Imc
lonG
Into
loou
6-6
1-40
Deant Dked
He har
Dand cut Eaickyy
Figure 9:
The effect of the number of semantic modulation blocks. We analyze 0, 4, and 10 semantic modulation blocks and choose to use 4 blocks.
Table 2:
Quantitative ablation analysis results.
Editing Performance
Text Prompt
w/o S
w/o
w/o S&T
Ours
== 0
k
k ==
= 6
k: == 10
Bald
0,0630
0.2391
0.2227
0.0279
0.2689
0.2371
0.0279
0.0236
0,1720
Angry
0.7510
0.7470
0.7640
0,5810
0.0669
0,5900
0.7380
0.6206
Rod hair
0.7082
0.7514
0.7769
0.7171
1.1566
0.7233
0,7171
0.6860
0.7515
Aged 10
13.9412
20.0762
15.8426
9.8874
18.8625
18.5182
9.8874
13.4156
14.9789
Effect of the Number of Semantic Modulation Block. To ablate the number of modulation block (k in Fig. 2), we train models with = 0, 2, 4, 6, 10 respectively and evaluate their performance. The visual results are shown Fig. 9. When mappers in StyleCLIP to train our model, and we find that the visual results cannot reflect the text semanties. few modulation blocks will result in poor editing (see k 2). Increasing the number of modulation blocks can improve the performance (k = 6), but larger & will cause unstable training and lead to higher computational the performance and efficacy. The numerical analysis results are shown in Table 2.
set the number of modulation blocks as
use the
(see k 10). We finally select
in our method balance
5 Concluding Remarks
We propose to manipulate image content according to text prompts with different semantic meanings. Our motivation is that latent mapping of existing approaches is empirically designed between text prompt embedding and the visual latent space, so one editing model tackles only one text prompt. We improve latent mapping by semantic modulations with alignment and injection. It benefits one editing model to tackle multiple text prompts. Experiments in multiple datasets show that our FFCLIP effectively produces semantically relevant and visually realistic results.
limitation of our method is that we do not completely disentangle the StyleGAN latent code W+, which still remains an open problem for all the existing methods. The CLIP is known to encode humanlike biases [52, 7]. As our text embedding is from CLIP, the manipulated results may also suffer from humanlike biases. Such biases may propagate and have negative effects minority representations (i.c., skin color changes drastically after editing). To overcome this limitation, we can use the text-inversion [12] to find the proper text embedding without biases, we will try this in future work. Moreover, FFCLIP generalizes when the training and testing text prompts share similar image attribute descriptions. It cannot change hair color if no hair-related prompts are utilized during training. The generalization of other encoders shown in Appendix remains for further investigations. The negative societal impact is that the edited human portraits could be used for malicious purposes.
Acknowledgements and Disclosure of Funding. We appreciate the insightful suggestions from the anonymous reviewers to further improve our paper. This work was supported by SZSTC Grant 17--CCYY2201908091222000 and WDZC20208220000555000 ZDSYYS200100222092000000 (Joint Research Center of Tencent and Tsinghua).
Shenzhen Key Laboratory
10

---

References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka, Image2stylegan: How to embed images into the stylegan latent space? In IEEE/CVF International Conference on Computer Vision, 2019.
[2] Rameen Ahdal, Yipeng Qin, and Peter Wonka. Image2stylegan+4: How to edit the embedded images? In IEEE/CVF Conference Computer Vision and Puttern Recoonition, 2020.
[3] Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. Styleflow; Attribue-condditooned exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics, 2021.
[4] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Only matter of style: Age transformation using style-based regression model. ACM Transactions on Graphics, 2021.
[5] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Restyle: A haidual-based stylegan encoder via iterative refinement. In /EEF/CVF International Conference on Computer Vision, 2021.
[6] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun- Jun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727, 2020.
[7] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe, Multimodal datasets: misogyny, pornog- raphy, and malignant stereotypes. arXiv preprint 10.00921 10.01963, 2021.
[8] Jianbo Chen, Yelong Shen, Jianfeng Gao, Jingjing Liu, and Xiaodong Liu. Language- based image editing with recurrent attentive models. In IEEE/CYF Conference on Computer Vision and Pattern Recognition, 2018
[9] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos 7afeiriou. Areface: Additive angular margin loss for deep face recognition. In IEEF/CVF Conference Computer Vision and Pattern Recognition, 2019.
[10] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang. et al. Cogview: Mastering text-to-imass im-ge generation via transformers. Advances in Neural Information Processing Systems, 2021.
[11] Jun Fu, Jing Liu, Haijic Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In IEEE/CVP Conference Computer Vision and Pattern Recognition, 2019.
[12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.0 01618, 2022.
[13] Chongjian Ge, Youwei Liang. Yibing Song, Jianho Jiao, Jue Wang, and Ping Luo. Revitalizing cnn attention via transformers self-supervised visual representation learning. Advances in Neural Information Processing Systems, 2021.
[14] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 2014.
[15] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. arXiv preprint arXiv:2111. 2021.
[16] Erik Härkönen, Aaron Hertzmann. Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems, 2020.
[17] Xianxu Hou, Linlin Shen, Or Patashnik, Daniel Cohen-Or, and Hui Huang. Feat: Face editing with attention. arXiv preprint arXiv:2202.02713. 2022.
[18] Xianxu Hou, Xiaokang Zhang, Hanbang Liang, Linlin Shen, Zhihui Lai, and Jun Wan. Guidedstyle: Attribute knowledge guided style manipulation for semantic face editing, Newral Networks, 2022.
[19] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:/710.10199 2017.
[20] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems. 2020.

---

[21] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[22] Tero Karras, Samuli I Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[23] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[25] Cheng-Han Hanng-HanLLa Lee, Ziwei Liu. Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[26] Heyi Li, Jinlong Liu, Yunzhi Bai, Huayan Wang. and Klaus Mueller. Transforming the latent space of stylegan for real face editing. arXiv preprint arXiv:2105. 14230, 2021.
[27] Wenbo Li, Pengehuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-driven text -o-ttoo-mage synthesis via adversarial training. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2019.
[28] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint X21:2202.007000, 2022.
[29] Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 2021.
[30] Hongyu Liu, Ziyu Wan, Wei Huang. Yibing Song, Xintong Han, and Jing Liao. Pd-gan: Probabilistic diverse gan for image inpainting. In IEFF/CVF Conference on Computer Vision and Pottern Recognition, 2021.
[31] Hongyu Liu, Ziyu Wan, Wei Huang. Yibing Song. Xintong Han, Jing Liao, Bin Jiang, and Wei Liu. Deflocnet: Deep image editing via flexible low-level controls. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[32] Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. Text-adaptive generative adversarial networks: manipu- lating images with natural language. In Advances in Neural Information Processing Systems, 2018.
[33] Patashnik. Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In IEFF/CVF International Conference on Computer Vision, 2021.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[35] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612 2022.
[36] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, 2021.
[37] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lee. Generative adversarial text to image synthesis. In International Conference on Machine Learning, 2016.
[38] Elad Richardson. Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: stylegan encoder for image-to-image translation. In IEEE/CVP Conference on Computer Vision and Pattern Recognition, 2021.
[39] Rasmus Rothe, Radu Timofte, and Luc Van Gool. Dex: Deep expectation of apparent age from a single image. In JEEE/CVF International Conference on Computer Vision Workshops, 2015.
[40] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020.
[41] Yujun Shen and Bolei Zhou Closed-fore factorization of latent semantics in gans. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
12

---

[42] Alon Shoshan, Naday Bhonker. Igor Kviatkovsky, and Gerard Medioni. Gan-control: Explicitly controllable gans. In IEEE/CVF International Conference on Computer Vision, 2021.
[43] Henrique Siqueira, Sven Magg. and Stefan Wermier. Efficient facial feature learning with wide ensemble- based convolutional neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, 2020.
[44] Ming Tao, Hao Tang. Songsong Wu, Nicu Sebe, Xiao- Jing, Fei Wu, and Bingkun Bao. Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint 20Xiv2008.00886. 2020.
[45] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Berard,, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[46] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics, 2021.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 2017.
[48] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In International Conference on Machine Learning, 2020.
[49] Binxu Wang and Carlos R Ponce. The geometry of deep generative image models and its applications. arXiv preprint arXiv:2 2021.
[50] Tengfei Wang, Yong Zhang. Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity gan inversion for image attribute editing. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2022.
[S1] Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Zhentao Tan, Lu Yuan, Weiming Zhang, and Nenghai Yu. Hairclip: Design your hair by text and reference image. arXiv preprint 12X15221 2021.
[52] Robert Wolfe and Aylin Caliskan. Markedness in visual semantic ai, arXiv preprint htt.:2005./1378 2022.
[53] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[54] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Atingan: Fine-grained text image generation with attentional generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[55] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of a large-scale image datasct using deep learning with humans in the loop. arXiv preprint 1xXiv:1506.03366 2015.
[56] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang. Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In IEEE/CVF International Conference on Computer Vision, 2017.
[57] Han Zhang. Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang. Xiaolei Huang, and Dimitris N Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks, JEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
[58] Zizhao Zhang, Yuanpu Xie, and Lin Yang. Photographic text-to-image synthesis with a hierarchically- nested adversarial network. In IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[59] Hengshuang Zhao. Jianping Shi, Xiaojuan Qi. Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [No] The code is proprietary now, and we will release our source code. The data and instructions are shown in 4.1
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.1.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets... (a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No]
14


Reviews:
[{"review_id": "u8F0fL41qvF", "reviewer": "Reviewer_7Kr6", "summary": "The paper proposes an attention-based mechanism for improving CLIP- based StyleGAN editing models. \n\nSpecifically, the paper takes the ‘latent mapper’ approach, where a neural network is trained to predict a latent-code change that would modify specific image attributes described through text. It argues that existing methods are trained to tackle specific texts, and therefore do not need to be aligned with CLIP’s space, limiting their ability to tackle novel texts.\n\nThe authors propose to better align the spaces through an attention mechanism, where the CLIP-text embedding is used to predict both layer and channel importance for latent code changes. This approach allows the authors to train their mapper on a larger collection of texts, creating better alignment between the CLIP and StyleGAN spaces, and enabling more disentangled modifications, with some generalization to unseen texts (e.g. combinations of texts seen during training).", "questions": "1) How well does the model generalize to unseen texts? Can a model not trained on hair-related prompts be used to change hair color? What if the only hair-related prompt was ‘blond hair’, would it still work to create red hair? What if the texts contained only ‘blond hair’ but also ‘red beard’, would compositionality in this manner work? If the model truly aligned the CLIP and StyleGAN spaces, one would expect this to be possible.\n\n2) I’m a bit surprised that StyleCLIP fails so badly on hair color changes. Their official models have a purple hair mapper which works fine, and I just tried the global directions in their notebook and was able to generate ginger hair. Is there any chance that you trained their mapper with the fine-layers disabled? \n", "limitations": "The limitations and social impact section of the paper is extremely bare-bones, consisting of 2.5 lines. \n\nA few likely limitations might be things discussed in my questions above: How well does the model generalize beyond the training text? \nAnother limitation might be the fact that you train specifically on e4e codes. What if better encoders come along and now, I want to use those? Can you generalize to latent codes derived through optimization? Using another encoder? \n\nOn the matter of social impact, this type of model can lead to harm in more than just the ability to create fake imagery. It can also propagate biases or have negative effects on minority representations. For example, in your Fig 6, the top-right individual’s skin color changes drastically after editing. It is fine that your model has these shortcomings, but the purpose of the limitations / impact paragraph is to be upfront about them.\n", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper introduces a conceptually simple yet well motivated approach for identifying the StyleGAN layers and channels most relevant to the change of specific attributes described through text – specifically, attention.  \n2) While there are prior works which deal with attention mechanisms in the context of StyleGAN inversion and editing [1, 2], they are, so far as I am aware, unpublished. However, I would urge the authors to cite these as related work (and consider comparisons to [2], which tackles the same task).\n3) The results are generally of superior quality when compared to existing approaches. In particular, they are more disentangled, and allow for more complex manipulations than the baselines.\n\nWeaknesses:\n\n1) Clarity: The paper itself is very difficult to follow. I had a very hard time understanding the abstract and the introduction, and had to delve into the method and the experiments themselves to get any idea of what the authors are trying to do. The paper is not so complex that this should be required. I urge the authors to, at the very least, run their paper through Grammarly or similar tools.\n\n2) Justification or experimental verification of claims: The core paper contains many claims which are unsubstantiated. In many cases, the supplementary **does** contain experiments which, at least in part, provide backings for these claims. However, they are never referenced from the core text, and if I did not read the appendix I would not be aware of them. These should absolutely be included in the main paper. A few examples:\n\n    * The authors claim time and again that their model aligns the CLIP and StyleGAN spaces. The only experiment which possibly hints at this is the interpolation experiment in the appendix. Other than that, it appears that all experiments use the same, or compositions of the texts used during training. \n\n    * It is unclear from the paper itself whether the benefit of their approach is a result of the attention mechanism (i.e. improved latent mapper architecture), or the fact that they train for multiple prompts. There is one experiment in the appendix that gives this some backing (Figure 7, the w/o S&T experiment). However, it would also be interesting to see that the original latent mapper can’t be trained conditionally on multiple prompts.\n\n\n3) Some (minor) wrong claims:\n   * In the paragraph of L107, the authors suggest that inversion is done into W+ because it is better for editing. This is not the case. Inversion is done into W+ because it allows for better identity preservation. W+ inversions typically behave worse under editing than their W counterparts. See for example e4e [3].\n   * The authors claim that StyleCLIP needs to train a different model for each textual prompt. This is only true for their latent mapper approach, but is not actually specified anywhere in the current submission. The ‘Global Directions’ method, for example, does not suffer from this limit. Actually, I would clarify early in the paper that this method specifically aims to improve on the latent mapper approach of StyleCLIP (this also ties to clarity above).\n\n[1] Transforming the Latent Space of StyleGAN for Real Face Editing, Li et al, 2021\n\n[2] FEAT: Face Editing with Attention, Hou et al, 2022\n\n[3] Designing an Encoder for StyleGAN Image Manipulation, Tov et al, SIGGRAPH 2021\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Discrimination / Bias / Fairness Concerns"]}, "extra_text": {}}, {"review_id": "mnfNif6bRBL", "reviewer": "Reviewer_rJnY", "summary": "In this paper, an image manipulation method FFCLIP is proposed to edit image semantics based on the text prompt guidance. Following the StyleGAN latent space W for image generation, the alignment between semantics in the latent space W and text encoder is crucial for text-driven image editing. A semantic modulation method is developed to align the visual representations and text semantic embeddings via linear transformation where the parameters are computed based on cross attention computations. This effective alignment improves the previous empirical alignment design so that text prompts with different semantics can be taken for image editing via one model, while the empirical alignment confines each text prompt corresponds to one model. Experiments show that for each dataset (portraits, cars, or churches images), using one model is able to free-formly edit images based on different text prompts. The supplementary videos have also shown the effectiveness.", "questions": "See weakness. A clear discussion upon previous text-driven methods and more recent diffusion models would strengthen the proposed method.", "limitations": "As illustrated in Ln 271, the disentanglement of styleGAN latent space is an open problem. The negative society impact is for face malicious, which can be mitigated via deepfake detection methods.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The alignment is effective to automatically correspond text prompt embeddings to StyleGAN latent Space. This enables different semantical embeddings to be reflected in the visual image representations for manipulations. \n+ The semantic modulation first aligns text-image semantics via semantic alignment and then injects text prompts semantic. This design is reasonable to modify StyleGAN latent space since this space is not disentangled completely at present. Without latent space disentanglement, using the alignment via cross attention computation matches the text-image semantics. \n+ Experiments on three benchmark datasets show that under each dataset, using one model is able to edit image semantics free-form based on different text prompts with single or multiple semantics. The visual results are impressive on both paper and the supplementary video. The numerical results also show the effectiveness of FFCLIP.\n\n- The existing methods TediGAN, StyleCLIP, HairCLIP also edit images based on the text information. A more in-depth analysis shall be taken to enhance the proposed contributions. For instance, TediGAN seems to tackle image editing with different text attributes. StyleCLIP uses one model for a specific text-image editing. HairCLIP focus on hair and can handle different text semantics. These would strengthen why the proposed FFCLIP makes senses and contribute to the cross-modal image manipulation area.\n- A discussion with diffusion models would be helpful to emphasize the importance of editing StyleGAN latent space to benefit future research.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_lJ6QFQ-x0J", "reviewer": "Reviewer_TtwX", "summary": "This paper proposes a new architecture for using CLIP to modify images in coordination with StyleGAN. The system is known as FF-CLIP (Freeform CLIP) and incorporates semantic alignment and injection blocks to identify a semantic subspace in the GAN embedding space. Cross-attention is employed to achieve semantic alignment, while an injection technique previously used in HairCLIP is employed to inject the text embedding semantics into GAN space. The authors provide visual examples of image transformations and provide quantitative evaluations by human subjects indicating that humans prefer FF-CLIP modified images.", "questions": "Does model size of the CLIP used make any difference? The larger CLIPs do much better on evaluations like ImageNet, so maybe there would be some benefit to using them here?", "limitations": "CLIP is known to encode humanlike biases (see Wolfe et al. 2022, ACM FAccT; Birhane et al. 2021, arXiv). It would be good to mention these in a limitations section. How might they affect the system being developed?", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality\nStrengths: The authors choose a relatively novel area of inquiry, that of providing text based guidance to a synthetic image generator, in this case a GAN. This is an area of much innovation at the moment.\nWeaknesses: This seems very similar to other recent CLIP-based image modification systems, especially StyleCLIP. While the authors report outperformance of StyleCLIP in terms of human evaluation, it is difficult for this reader to tell that much of a difference between the StyleCLIP generated images and the FF-CLIP generated images based on the examples provided. The semantic injection technique has itself been used previously in HairCLIP.\n\nQuality\nStrengths: It is an interesting paper and to my knowledge the alignment/injection architecture is novel and might be useful.\nWeaknesses: Evaluations are not robust enough to be convincing to this reader: the authors evaluate only six image modifications based on human preferences. It’s hard to know whether the results are significantly better than those achieved by, e.g., StyleCLIP when the evaluation is narrow.\n\nClarity\nStrengths: Good visualizations, and the math is presented in a straightforward and comprehensible manner.\nWeaknesses: The writing can be hard to follow, with grammatical issues throughout. \n\nSignificance\nStrengths: Synthetic image generation is gaining ground rapidly and this system may help to advance the state of the art.\nWeaknesses: The proposed system mainly consists of components previously used in very similar systems such as StyleCLIP and HairCLIP. If the evaluation of the system were more robust, this might not be an issue, but it is also hard to gauge how much of an advance this model is.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept