CLIP-PAE: Projection-Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable and Controllable Text-Guided Image Manipulation
Authors: Chenliang Zhou
Fangcheng Zhong
Cengiz Oztireli
Venue: Submitted to ICLR 2023 (2023)
Keywords: computer vision
text-guided image manipulation
latent manipulation

Abstract: Recently introduced Contrastive Language-Image Pre-Training (CLIP) bridges images and text by embedding them into a joint latent space. This opens the door to ample literature that aims to manipulate an input image by providing a textual explanation. However, due to the discrepancy between image and text embeddings in the joint space, using text embeddings as the optimization target often introduces undesired artifacts in the resulting images. Disentanglement, interpretability, and controllability are also hard to guarantee for manipulation. To alleviate these problems, we propose to define corpus subspaces spanned by prompts to capture specific image characteristics. We introduce CLIP projection-augmentation embedding (PAE) as an optimization target to improve the performance of text-guided image manipulation. Our method is a simple and general paradigm that can be easily computed and adapted, and smoothly incorporated into any CLIP-based latent manipulation algorithm to improve performance. To demonstrate the effectiveness of our method, we conduct several theoretical and empirical system studies. As a case study, we utilize the method for text-guided semantic face editing. We quantitatively and qualitatively demonstrate that PAE facilitates a more disentangled, interpretable, and controllable image manipulation method with state of the art quality and accuracy.

Under review as a conference paper at ICLR 2023
CLIP-PAE: PROJECTION-AGGMENTAAII DING TO EXTRACT RELEVANT FEATURES FOR A DIS- ENTANGLED, INTERPRETABLE, AND CONTROLLABLE TEXT-GUIDED IMAGE MANIPULATION
EMBED-
Anonymous authors Paper under double-blind review
ABSTRACT
Recently introduced Contrastive Language-Iman Pre-Training (CLIP) (Radford et al. 202I) bridges images and text by embedding them into a joint latent space. This opens the door ample literature that aims manipulate an input image by providing a textual explanation. However, due to the discrepancy between image and text embeddings in the joint space, using text embeddings the optimization target often introduces undesired artifacts in the resulting images. Disentangle- ment, interpretability, and controllability are also hard guarantce for manipula- tion. To alleviate these problems, we propose to define corpus subspaces spanned by relevant prompts to capture specific image characteristics. We introduce CLIP embedding (PAE) as an optimization target to improve the performance text-guided image manipulation. Our method a simple and general paradigm that can be easily computed and adapted, and smoothly incor- porated into any CLIP-based image manipulation algorithm. To demonstrate the effectiveness of method, conduct several thcoretical and empirical stud- ies. As a case study, we utilize the method for text-guided semantic face editing. We quantitatively and qualitatively demonstrate that PAE facilitates more disen- tangled, interpretable, and controllable image manipulation with taa quality and accuracy.
INTRODUCTION
In text-guided image manipulation, the system receives image and text prompt and is tasked with editing the image according to the text prompt. Such tasks have received high research attention due to the great expressive power of natural language. Recently introduced and increasingly popular Contrastive Language-Iman Image Pre-Training (CLIP) (Radford et al. 2021] is a technique to achieve this by embedding images and texts into joint latent space. Combined with generative techniques such as Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) and Diffusion (Ho et al.) 2020 Dhariwal Nichol 2021). CLIP has been utilized develop several high-quality image manipulation methods (e.g.. Ling et al. 2021| Zhang et al. 2021 Antal & Bodō 2021 Khodadadch
al.
2022). where the image optimized be similar to the text prompt in the CLIP joint space.
There are three important but difficult to satisfy properties when performing image manipulation: disentanglement, interpretability, and controllability. Disentanglement means that the manipulation should only change the features referred to by the text prompt and should not affect other irrelevant attributes (Wu al. 202T, Xu et al.] (022). Interpretability means that we know why/how an edit to the latent code affects the output image and thus understand the reasoning behind each model decision (Doshi-Vell Velez & Kim 2018) Miller 2019). or that the model can extract relevant information from the given data (Murdoch control the intensity of the edit (You et al., 2021 Park et al. 2020 Li et al. 2019) for individual factors and hence is tightly related to disentanglement.
al. 2019). Finally, controllability is the ability to
In CLIP- text texted image manipulation methods, since both the latent space of the generative network and the embedding space of CL.IP extensively compress information (e.g., 1024 x 1024

---

Under review as a conference paper at ICLR 2023
image contains 3 x 1024 dimensions, whereas typical StyleGAN (Karras et al. 2019) and CLIP both only have 512-dimensionnl latent spaces), the manipulation in the latent space is akin to a black box. (Gabbay et al. (202I) argue that GAN fails at disentanglement because it only focuses on localized features. Zindancioglu & Sezgin (021) also show that several action units in StyleGAN (Karras et al. 2019) are correlated even if they are responsible for semantically distinct attributes in the image. In addition, we found that although CLIP embeddings of images and texts share the same space, they actually reside far away from each other (see Section 3.1), which can lead to undesired artifacts in the generated images such as unintended editing or distortion of facial identity (see Figure [ Finally, most existing methods for CLiP-based text-guided image manipulation do not allow for free control of the magnitude of the edit. As result, how to perform image manipulation in a disentangled, interpretable, and controllable way with the help of text models remains a hard and open problem.
In this paper, we introduce a technique that can be applied to any CLIP-based text-guided image manipulation method to yield more disentangled, interpretable, and controllable performance. Rather than optimizing an image directly towards the text prompt in the CLIP joint latent space, embedding (PAE), as the
propose novel type of CLIP embedding, rngjection-auugmenaaatiio optimization target. PAE was motivated by two empirical observations. First, the images do not overlap with their corresponding texts in the joint space. This means that text embedding does not represent the embedding of the true target image that should be optimized for. Therefore, directly optimizing an image to be similar the text in the CLIP space results in undesirable artifacts or changes in irrelevant attributes. Second, a CLIP subspace constructed via embeddings of set of relevant texts can constrain the changes of an image with only relevant attributes.
To construct a PAE, we first project the embedding of the input image onto a corpus subspace constructed by relevant texts describing the attributes we aim disentangle, and record residual vector. Next, we augment this projected embedding in the subspace with the target text prompt, allowing for a user-specified augmenting power to control the intensity of change. Finally, we add back the residual to reconstruct vector close to the "image region" of the joint space. We demonstrate that the PAE is a better approximation to the embedding of the true target image. With PAE, we achieve better interpretability via an explicit construction of the corpus subspace. We achieve better disentanglement since the subspace constrains the changes of the original image with only relevant attributes. We achieve free control of the magnitude of the edit via user-specified augmenting power. PAE is easy to use, quick to pre-compute, and can be incorporated into any CLIP-based latent manipulation algorithms to improve performance.
In short, we highlight the three major contributions of this paper:
1. (Section 3
We perform a series of empirical analyses of the CLIP space and its subspaces, identifying i) the limitations of using naive CLIP loss for text-guided image editing; and ii) several unique properties of the CLIP subspace.
2. (Section Based on our findings in Section 3 we propose the
embedding (PAE) as a better approximation to the embedding of the true target image.
3. (Section 5) We demonstrate that employing PAE an alternative optimization target facil- itates a more disentangled, interpretable, and controllable text- text-guided image manipulation. This is validated through several text-guided semantic face editing experiments where integrate PAE into a set of state-of-the-an models. We quantitatively and qualitatively demonstrate that PAE boosts the performance of all chosen models with high quality and accuracy.
2 RELATED WORK
Latent Manipulation for Image Editing One popular approach for image manipulation is based
on its latent code: the input image is first embedded into the latent space of a pre-trained generative
network such as GAN (Goodfellow et al. 20144), and then to modify the image, one updates either
the latent code of the image, (e.g., Zhu et al., 2016:] Ling et al., 2021] Zhang et al. [021 Antal &
Bodó
2021
Khodadadeh al. 2022] Creswell & Sharath|22015 Perarnau al. [0166] Pidhorskyi
ct al.
2020
Hou et al.] 2022b] Xia et al.) 2021] Kocasari et al., 2022) Patashnik et al. [021] Shen
al.
2020), or the weights of the network (e.g.. Cherepkov al. 2021
Nitzan et al.
2022 Reddy
2

---

Under review as a conference paper at ICLR 2023
et al. 2021) to obtain the desired image editing. However, these methods can only alter set of pre-defined attributes and thus lack flexibility and generalizability.
CLIP for Text-Guided Image Manipulation In 2021 Radford et al.) proposed Contrastive Language-Image Pre- Pre-Training (CLIP), where an image encoder and a text encoder are trained such that the semantically similar images and texts also similar in the joint embedding space. The insight of connecting image and text in the same space brings up a wide spectrum of applications in computer vision tasks. For example, Radford al.] provide applications for image captioning, im- age class prediction, and zero-shot transfer. More sophisticated tasks include language-driven image generation (Ramesh al.
2022). zero-shot semantic segmentation (Li et al. 2022), image emotion
classification (EEC) Deng et al. 2022), large-scale detection of specific content (González-Pizze Pizarro
& Zannettou 2022). object proposal generation (Shi al. 2022b), object sketching (Vinker al.)
2022 and referring expression grounding (Shi et al. 2022a).
CLIP has also been applied to text-guided image manipulation tasks. In this domain, one approach is to edit the latent code of certain generative network so that the embedding of the generated im-
age is similar to the embedding of the given text in the CLIP space (Kocasari et al.) | 2022 Patashnik et al. 2021 Xia et al.) 2021] Hou et al. 2022a. Ramesh et al. (022) (see Figure 4a). However, this straightforward approach sometimes fails to change the desired attributes, or fails to change them in disentangled way: other unrelated features are also affected (see the comparative study in Sec- tions an3ddd33) In addition, the proposed methods often fail to exhibit enough interpretability and controllability. This is probably attributed to the separation of images and texts in the CLIP embed- dings space, so that optimizing an image embedding towards a text embedding naturally introduces some undesired effects. Although there are some remedies, such as penalizing the change of features in the editing (e.g., (Canfes et al. 2022 Kocasari et al. 2022)) ), separating the image into different granule levels (Patashnik et al.J 2021), formulating a constrained optimization problem (Zhu al., 2016). partially labeling a set of features (Gabbay et al. (2021), or updating the parameters of the underlying GAN to preserve features (e.g.. (Nitzan et al. 2022 Reddy et al., 202T Cherepkov et al.) 2) they do not collectively achieve disentangled, interpretable, and controllable manipulation. Additionally, some of them target all features indifferently: they still do not separate the related fea- tures from the irrelevant ones. Moreover, most of these techniques only work for specific methods or attributes and lack generalizability, and some of them are too complicated and time-consuming. hence cannot be used in practical large-scale applications.
3 ANALYSIS OF THE CLIP JOINT SPACE AND SUBSPACE
Most HLIP-based text -uxidd image manipulation algorithms (e.g.. Kocasari et al. 2022 Patashnik
et al.) 2021: Xia et al. 2021 Hou et al. 2022a Ramesh et al.) 2022 follow general paradigm
where certain parameters of an image editing process (such as the latent code or the weights of a generative network) are trained minimize casine similarity loss between the resulting image and a text prompt in the CLIP joint space (see Figure 4a). However, naively using this loss may introduce undesirable artifacts or unintended changes unrelated the text prompt, as shown in Figure We hypothesize that this is due to the discrepancy between the images and the texts in the CLIP joint space, as we will demonstrate in Section 3.1 Therefore, the embedding of the text prompt in the CLIP joint space does not actually represent the embedding of the true target image that should be optimized for. To alleviate this issue, we construct CLIP subspaces with desirable properties (see embedding (see Section
Section 3.2 3.2 leading to our proposed
3.1 NON-OVERLAPPING IMAGE AND TEXT EMBEDDINGS
As an empirical demonstration, we collect over 1500 face images and 1500 textual descriptions of faces (e.g., emotion, hairstyle or general) and visualize their embeddings in the CLIP joint space using PCA (Jolliffe] 2002) in Figure 2 Note that the visualization is given using Euclidean distance; Although CLIP is trained with cosine similarity, if we normalize all the embeddings, the Euclidean distance and cosine distance give exactly the same ranking because
||a b|22 V(a b)2 V a² b2 Ta /2 2CosSim(a, b).
(1)
As shown in Figure 2 the image and the text embeddings lie in two non-overlapping regions. They also exhibit lower inter-modality cosine similarity compared to the intra-modality similarity,
3

---

Under review as a conference paper at ICLR 2023
Original
0.4
0.2-
0.0
happy
-0.4
0.4
-0.2
0.0
0.2
0.4
0.6
face images face texts
emotion texts
Ansotton PAE
black hair
hairstyle texts
Figure PCA visualization of face images and text descriptions (Section [. and their corresponding PAEs in the CLIP space (Sec- tion 4.. Note that image and text embeddings are non-overlapping. that PAEs are closer to image, and that PAEs spread less widely than the face images, indicating that they contain more specified information.
StyleCLIP
StyleMC
TediGAN
Using text embedding the opti- mization target results in unsatisfactory out- puts. Note the entangled changes (e.g., cloth- ing of the left child, glasses of the lady), in- accurate output, and artifacts.
Figure/ 1:
gardless of their semantic meanings. For example, the cosine similarity between an image of a dog and the text "dog" is 0.253 and that between an image of a cat and the text "cat" is 0.275. On the other hand, the similarity between a dog image and a cat image reaches 0.841, and that between the two texts "dot" and "cat" is as high as 0.936, much higher than the intra-modality similarity. We record additional evaluations of inter-modality and intra-modalit, cosine similarities in Appendix [A.] In the general paradigm described above (and see Figure 44), the CLIP embedding of the ideal target image is essentially approximated by the embedding of the text description. However, the separation of text and image embeddings clearly invalidates such approximation and thus leads to artifacts (see
Figure I and more in Sections 5.2 and 5.3)
3.2 SUBSPACES DISTILLING RELEVANT INFORMATION FROM IMAGE EMBEDDINGS
Since the joint space is vector space, we construct subspace of it using a set of relevant text prompts as basis vectors. For example, we can construct an "emotion subspace" using relevant emotion texts. In this section, we explain how such subspace distills relevant information from images, which can be used to constrain the changes of an image. We also include some experiments in Appendix A.5 to show that the subspace can distill information from texts. We leave the mathe- matical details of the formulation of the subspace to Section 4 but preview certain properties of the subspace as these properties inspire the formulation of our method.
We invited ten participants to record five-second videos of their face changing from neutral to laugh- ing out loud (LOL). We compute the cosine similarity (averaged over all videos) of each frame to the first frame or to the text "a happy face" in the CLIP space, or after projecting them onto the emotion or hairstyle subspace. The results are shown in Figure
In Figure Ba the similarity decreases much faster in the emotion subspace compared to the CLIP space and the hairstyle subspace. A similar pattern shows up in Figure 33 the similarity in- creases much faster in the emotion subspace. Since the subspace has a lower dimensionality than the original space, these observations indicate that the information regarding the irrelevant attributes is discarded during the projection. Hence the changes of the hairstyle in the facial images have low effect influencing the similarity - the emotion subspace only distills emotional attributes from the image and discards others. This inspires the formulation of our method: if we make changes to the embedding of the original image within a subspace, it would only induce a change of attributes related to the subspace. As result, these changes are disentangled.
4

---

Under review as a conference paper at ICLR 2023
0.8
1.00
0.7
Similarity
0.6
in CLIP space
0.85
0.5
In emotion subspace
0.80
in hairstyle subspace
0.4
in CLIP space
0.70
emotion subspace hairstyle subspace
0.3
0.65
25
50
75
100
125
0
25
50
75
100
125
Frame (0- neutral to 135-LOL)
Frame (0-neutral 135-LOL)
(a) Similarity to the first frame
(b) Similarity to the text "a happy face"
Figure 3: Similarity of video frames to the first frame and to the text "a happy face" in different spaces. The changes in the emotion subspace are the most significant as it distills the relevant information.
4
EMBEDDING
Motivated by our findings in Section we propose the better approximation the embedding of the true target image. There are three objectives aim to fulfill when constructing such an embedding: i) it should be closer to the image region than the text is; ii) should be guided by the target text prompt: iii) the guidance should be provided within the subspace so the changes are disentangled.
embedding (PAE)
4.1 OVERVIEW
Given an input image I and text prompt T, we construct the PAE, denoted as Exn a), as follows. First, we obtain the embeddings of and T, denoted as e, and er, via the CLIP image and text encoders (Radford et al. 2021), then we project e, onto a corpus subspace 2T constructed with texts describing the attributes aim disentangle:
(en)
where Par is the projection operation onto the subspace WT We will explain the details of Pan and the construction of 2I in Section 4.2 Next, we record residual vector r of the projection:
-1 w.
After that, augment the influence of the text prompt T to the projected embedding and finally, add back the residual vector r to construct the final embedding:
Egn (I, a.)) Am, ax) +
where Ag,, T the augmentation operation in 20 according T' with a controllable augmenting power a, as will be explained in details in Section
A graphic illustration of PAE can be found in Figure objectives, we highlight that we add back the residual to ensure that the PAE is close to the image region rather than the text region, that we apply the augmentation to ensure that the PAE guided by the target text prompt, and that we apply the projection to ensure that the guidance is provided via the subspace so the changes of the image are disentangled (recall Section (.2). As we will see in Section 4.2] the construction of the subspace with explicit selections of relevant prompts makes our approach more interpretable compared to black-box latent manipulations. Our introduction of the augmenting power (see Section 4.3) makes it possible freely control the magnitude of the change to the image. PAE can be integrated into any CLIP-based text-guided image manipulation algorithms in place of the text embedding in the final loss function, shown in Figure 4b
40 In response our aforementioned three
In Figure 2 we also include corresponding PAES generated by the same face images and facial descriptions in Section 3.1 We that compared to the text embeddings, PAES are indeed closer
5

---

Under review as a conference paper at ICLR 2023
Tlepes"
C1.7 mowie
axbeelinng
miiteliing
embedded image
image region
axcid
residual
(a) A general paradigm followed by most CLIP- based text- guided image manipulation algorithms for semantic face editing (c.g.. Patashnik et al.
Pied
A(P(R))
2021
Kocasari et al. 2022 Xia et al. 2021).
augmenting direction
subspace 213
Costin
CLIP space
Tagps"
A graphical demonstration of the calculation of PAF. It is calculated by 1. projecting the em- bedded image e, (black) to a pre-defined sub- space 20 of interest (red); 2. augmenting the pro- jected vector in way that the effect of text is am- plified (blue): and finally 3. adding back the resid- ual r to return to the "image region" in & (purple).
a1 Lasge-olitne cal.te--littgg. proccc Inanelle Inanell secattiern
CLIP Iooge
Iopss Image
(b) Integration of into the CLiP-based text- guided image manipulation paradigm.
Figure 4
to the image embeddings, and that the PAE distribution has lower variation, suggesting that they contain more specified information.
In the following subsections, we introduce the details of the projection and augmentation operations.
4.2 PROJECTION
We introduce two options for the projection operation: P and PPCA, PaS aims to find the semantic vectors for 20 the subspace semantically meaningful structure. For example, if we aim to change/disentangle the facial emotions, we can use the text embeddings of a set of basic emotions as the basis vectors.
20
After selecting the basis vectors }b, we apply the mal basis {6,) Jn. and then project e, onto 20 by computing a dot product with the basis vectors:
process to obtain an orthonor-
Pus (ei) Σ 6f
by-
PGS will fail if there is no apparent semantic basis (e.g., for hairstyle, it is hard to find "basic hairstyles"). PRCA aims to find the basis of 21 by constructing a set T consisting of a corpus of relevant text embeddings and performing principal component analysis (PCA) (Jollifee (002)) to extract a pre-defined number of principal components as the basis [b Effectively, the space spanned by [b, would be space approximating 217 that encompasses all the related texts the corpus of interest. After we get the basis, the projection is performed the same way as in Eq. (5). Other dimension reduction techniques can also be used in replace of PCA, such as kernal PCA (Schölkopf et al. 1997) or t-SNE (Van der Maaten & Hinton 2008)).
We also experimented with simpler idea of projecting both e, and er onto 20 and perform the op- timization in 20, instead of augmenting in 20 and adding back the residual. However, this approach introduced significant artifacts and entangled changes. possibly due the loss of information during the projection (see Appendix A.4 for more details).
4.3 AUGMENTATION
The augmentation operation directs embedding of the original image towards the target. As dis- cussed above, the augmentation on the projected image should be guided by the target text prompt.

---

Under review as a conference paper at ICLR 2023
A simple augmentation operation can be
A A a) = w + aer.
(6)
However, in our pilot studies, we found that this simple operation results in PAE too similar to the original embedding, leading to optimization process barely doing anything (see Appendix AA.3 We strengthen the influence of er on w by weakening the unintended attributes of w - components of where Pau (er) has a low value while preserving the sum of coefficients of w.
Mathematically, we first calculate the coefficients of w and the coefficients de of the projected text Pan (eT), expressed under the basis {bn] J 20 established in the previous section:
C₁ = w T bui
(7)
dg Pw (er((r)) (er) b
Next, we weaken (realized by subtraction) all components of and then add back the projected text Per (eT))). with coefficient such that the sum of coefficients is preserved so that the embedding does not deviate much:
Am, (w, a) = Σ (cx or (cxl) by
a
====1 Ick Pan (eT),
Σk=1 dx
1=
where R controls the augmenting power, contributing controllable latent manipulation, demonstrated in Figure 5b] The two terms in Eq. (9) are equivalent to weakening the components that are small in the projected text.
We also include two more options for projection and three more for augmentation in Appendix A.2 including the case when 21 is not linear subspace but a general manifold (so that there is no notion of basis). Future researchers can also develop more effective options suitable for their own specific tasks, and this is where lies the extensibility of our PAE, A preliminary criteria for evaluation and selection of different PAEs are present in Appendix A.3
5 EXPERIMENT
As a case study, we utilize PAE in a series of text-guided semantic face editing experiments. where we manipulate high-level (emotion) and low-level (hairstyle) facial attribute (Patashnik et al. 2021 Lyons et al.) (200).. We also show the results of manipulating other facial attributes such as size of the eyes and the size of mouth in Appendix A.6 and the results of manipulating non-facial images such as dogs in Appendix A.7
5.1 DATA AND PROCEDURE
Given randomly generated latent codes, we generate facial images using an implementation of Style- GAN2 (Karras al. 2020 with adaptive discriminator augmentation (ADA) and the pre-trained model weights for the FFHQ face dataset (Karras et al. 2017). We randomly generate initial latent codes, and then filter out invalid faces, leaving 28 images for the task.
As explained in Section 4 PAE can be integrated into any CLIP-based text-guided image manipu- lation algorithms improve performance. We integrate PAE into StyleMC (Kocasari et al. 2022), StyleCLIP (Patashnik et al. 2021). and TediGAN (Xia et al. 2021), comparing their performance with and without PAE. To further demonstrate the disentangling power of our method and ablate other influencing factors, we additionally apply PAE to a naive image manipulation algorithm we directly update the latent code of the randomly generated input images before the differentiable StyleGAN2 generator, according to either the naive loss (Figure 4aj loss with PAE (Figure 4b). Note that this approach does not include any additional supervision losses (such as an identity loss (Deng al. 2019)) other trainable parameters. As we will see in the following subsections, PAE can achieve identity preservation even without the identity loss.
In total, compare four pairs of models; each pair consists of the original version (Figure 4a its +PAE version (Figure [bb:: 1. Naive approach (Nv) vs. Naive+PAE (Nv+); 2. StyleMC (SM) Kocasari et al. 2022) vs. StyleMC+PAE (SM+); 3. StyleCLIP (SC) (Patashnik et al. 2021)
7

---

Under review as a conference paper at ICLR 2023
StyleCLIPPPAE (SC+); and 4. TediGAN (TG) TediGAN (Xia et al. 2021 TediGAN+PAE (TG+). Note that each pair naturally forms an ablation study for PAE. The improvement brought by the PAE can be observed by comparing each original model with its +PAE version.
To construct the PAES, for emotion editing, we use PGS emotion defined via six basic human emotions happy, sad, angry. fearful, surprised, and disgusted
the semantic basis; for hairstyle editing, we use hainstyle defined via 68 hairstyle texts and ten principal components. The augmenting power is automatically selected according to the criteria stated in Appendix A.3
(Ekman 1992)
5.2 QUALITATIVE COMPARISON
qualitative comparison shown in Figure 5a The text prompts are written to the left of each row. Furthermore, in Figure 5b) we fix the text prompt to be "happy" and vary the augmenting power o, showing the controllability of PAE. Please refer to Appendix A.6 for full results.
happy
surppit sed
bald
Original
Naive
Naive PAE
StyleCLIP
StyleCLIP +PAE
StyleMC
StylcMC +PAE
Tedi GAN
TediGAN + PAE
(a) Comparison of eight models in text-guided face editing. We see that PAE promotes disentangled editing.
a=3
(b) Varying the augmenting power a when making faces happy show the controllability
Figure 5: Experiment results
From the Figure, we can see that in the naive approach, changes are not made in a disentangled way. For example, the hair color and the lighting condition of the face in the first changed; the identity of the face in the second row changed; etc. These problems are also present in the other three original models. We can also that models before and after equipping with PAE, we see that PAE enables a much more disentangled, realistic and accurate face manipulation.
edits do not have high quality accuracy. Comparing
5.3 QUANTITATIVE COMPARISON
In Table 1 we measure the performance of the aforementioned eight models with quantitative comparison seven metrics: Fréchet Inception Distance (FID) (Heusel et al.) 2017) (measures the quality of manipulated images), Learned Perceptual Image Patch Similarity (LPIPS) (measures the perceptual similarity
manipulated images to the original ones), Identity loss (IL) (uses ArcFace (Deng al. (019) to measure the degree of facial identity preservation), disentanglement measure- ment with a facial-attrin atttibbee classifier (Dis-C) (uses a facial attribute classifier (Serengil & Ozpinar 2021] to evaluate the degree of disentanglement by measuring changes in the model classification of
8

---

Another limitation of our method is that the concrete projection and augmentation operations need manual selection for the best result. We automated a coarse selection by the criteria introduced in Appendix A.3] but rigorously defined, numeric metric for disentanglement, accuracy, etc. could benefit the performance. Also, texts that does not associate with an obvious attribute (e.g., Under review as a conference paper at ICLR 2023
irrelevant attributes before and after editing), accuracy measurement with facial classifier (Acc-C) (uses the same facial attribute classifier to measure the degree of conformity to the text prompt), disentanglement evaluation from survey (Dis-S) (measures the degree of disentanglement with a survey), and accuracy evaluation from survey (Acc-S) (measures the degree of conformity to the text prompt with survey). The first three scores computed over the editing result of 5880 randomly generated images. The middle two scores from classifiers are computed over 48 emotion editing images: Dis-C is calculated from the percentage of the model classification of irrelevant tributes (age and race) that remain unchanged after the manipulation; and Acc-C is calculated from the percentage of output images whose model-pe preeiicedd emotion the same as the text prompt. Fi- nally, the two scores from survey are obtained from user study involving 50 participants of various backgrounds evaluating the editing of 36 randomly generated images. We used for a in PAE in all methods.
Table I: Seven metrics for eight image editing models. The & besides a metric means that lower score is better and the 1 means the opposite.
Model
Nv
Nv+
SC
SC+
SM
SM+
TG
TG+
FID
96.322
73.061
75.684
67.800
73.522
63.161
79.784
63.203
LPIPS +
0.251
0.103
0.157
0.071
0.123
0.094
0.298
0.073
IL
0.279
0.140
0.198
0.134
0.144
0.119
0.618
0.232
Dis-C (%)
30.2
36.3
39.6
35.4
33.3
38.6
30.2
35.5
Acc-C (%) f
20.8
60.4
22.9
58.3
29.2
27.1
58.3
62.5
Dis-S f
5.2
18.3
5.6
18.8
6.5
20.2
1.5
13.1
Acc-S (%) 1
3.8
19.7
8.7
24.7
7.8
23.8
6.8
20.8
From Table we see that Nv+ already outperforms the four original models in all scores except for Dis-C, where it is second SC. More importantly,
that almost all scores of the four models improved after equipping with PAE. It is also worth noting that IL is included in the training loss in SC and SM but not Nv+, nevertheless Nv+ outperforms SC and SM in terms of IL. These observations strongly indicate that PAE can improve the image quality, identity preservation, disentanglement and accuracy in image editing tasks.
DISCUSSION, LIMITATION, AND FUTURE WORK
An interesting observation from our experiment in hairstyle editing is that the difficulty manipu- lation complies with the real-world situations. For example, all models find it very hard to change women's 's hairstyle to be bald whereas men are very easy to lose hair. This suggests that the perfor- mance of such image manipulation methods is limited by the real-world datasets that upstream models (StyleGAN2 and CLIP) are trained on.
a celebrity's name) are harder to project to the subspace.
7 CONCLUSION
In this paper, we considered the problem of Caae-based text-guided image manipulation. We troduced novel pnjjection-aggmentttion embedding (PAE) to facilitate a more disentangled, inter- pretable and controllable image editing. PAE a general technique that can be seamlessly integrated into any other CLIP-based models: it is simple to use and has the advantage of compute efficiency, extensibility and generalizability. We demonstrated its effectiveness in text-guided semantic face editing experiment at both high-level (emotion) and low-level (hairstyle) editing. We integrate PAE with other state-of-the-as CLIP-based face editing models, showing the improvements brought by PAE in various perspectives.

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
The results in the main part and the appendix of this paper can be reproduced using the source code submitted supplementary material together with this paper. The project web page as well as the repository will be made publicly available after the blind review process.
REFERENCES
László Antal and Zalán Bodó. Feature axes orthogonalization in semantic face editing, In 2021 IEEE 17th International Conference on Intelligent Computer Communication and Processing (ICCP), pp. 163-169. IEEE, 2021.
Zehranaz Canfes, M Furkan Atasoy, Alara Dirik, and Pinar Yanardag. Text and image guided 3d avatar generation and manipulation. arXiv preprint arXiv:2202.06077 2022.
Anton Cherepkov, Andrey Voynov, and Artem Babenko. Navigating the gan parameter space for semantic image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3671-3680, 2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wor Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8188-812 8197,, 2020.
Antonia Creswell and Anil Anthony Bharath. Inverting the generator of a generative adversarial network. IEEE transactions on neural networks and learning systems, 30(7):1967-1974 2018.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
Sinuo Deng, Lifang Wu, Ge Shi, Lehao Xing, and Meng Jian. Learning to compose diversified prompts for image emotion classification. arXiv preprint arXiv:2201. 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
Finale Doshi- Velez and Been Kim. Considerations for evaluation and generalization in interpretable machine learning. In Explainable and interpretable models in computer vision and machine learning, pp. 3-17. Springer, 2018.
Paul Ekman. An argument for basic emotions. Cognition & emotion, 6(3-4):169-204, 1992.
Aviv Gabbay, Niv Cohen, and Yedid Hoshen. An image is worth more than a thousand words: Towards disentanglement in the wild. Advances in Neural Information Processing Systems, 34: 9216-9228, 2021.
Felipe González-Pizznn Pizarro and Savvas Zannettou. Understanding and detecting hateful content using contrastive learningg arXiv preprint arXiv:2201.0838) 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Asso- ciates, Inc., 2020. URL httpl: http://eeiiiing..iiiii....... http://sws...aaaooooooo///////// fc55 papppe7799aaaapapp..oo.. pdd
10

---

Under review as a conference paper at ICLR 2023
Xianxu Hou, Linlin Shen, Or Patashnik, Daniel Cohen- Or, and Hui Huang. Feat: Face editing with attention. arXiv preprint arXiv:2202.02713, 2022a.
Xianxu Hou, Xiaokang Zhang, Hanbang Liang, Linlin Shen, Zhihui Lai, and Jun Wan. Guidedstyle: Attribute knowledge guided style manipulation for semantic face editing. Neural Networks, 145: 209-220, 2022b.
lan Jolliffe. Principal component analysis for special rypes of data. Springer. 2002.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv:1710.1019, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Siavash Khodadadeh, Shabnam Ghadar, Saeid Motiian, Wei-An Lin, Ladislau Bölöni, and Ratheesh Kalarot. Latent to latent: learned mapper for identity preserving editing of multiple face at- tributes syylegan-generatee images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3184-3192, 2022.
Umut Kocasari, Alara Dirik, Mert Tiftikci, and Pinar Yanardag. Styleme: Multi-channel based fast text-guided image generation and manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 895-904, 2022.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image genera- tion, Advances in Neural Information Processing Systems, 32, 2019.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201. 03546, 2022.
Huan Ling, Karsten Kreis, Daiqing Li, Scung Wook Kim, Antonio Torralba, and Sanja Fidler. Edit- gan: High-precision semantic image editing. Advances Neural Information Processing Systems, 34, 2021.
Michael Lyons, Julien Budynek, Andre Plante, and Shigeru Akamatsu. Classifying facial attributes using 2-d gabor wavelet representation and discriminant analysis. In Proceedings fourth IEEE international conference on automatic and gesture recognition (Cat. No. PRO0S80), pp. 202- 207. IEEE, 2000.
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intel- ligence, 267:1-38, 2019.
W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, meth- ods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, (16(44):22071-22088 2019.
Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or. Mystyle: A personalized generative prior. arXiv preprint arXiv:2203.17220 2022.
Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. Advances in Neural Information Processing Systems, 1177199772111 2020.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- on Computer Vision pp. 2085-2091 2021.

---

Under review as a conference paper at ICLR 2023
Guim Peramna,, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez. Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355. 2016.
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14104-14113, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Interaational Conference on Machine Learning, pp. 8774-87663 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
Ravi Kiran Reddy, Kumar Shubham, Gopalakrishnan Venkatesh, Sriram Gandikota, Sarthak Khoche, Dinesh Babu Jayagopi, and Gopalakrishnan Srinivasaraghavan. One-shot domain adaptation for semantic face editing of real world images using stylealae. arXiv preprint arXiv:2/08. 13876, 2021.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Kernel principal component anal- ysis. In International conference artificial neural networks, pp. 583-588. Springer, 1997.
Sefik Ilkin Serengil and Alper Ozpinar. Hypercxtended lightface: A facial attribute analysis frame- work. In 2021 International Conference Engineering and Emerging Technologies (ICEET), pp. 1-4. IEEE, 2021. doi: 10.1 10.11099/CEE1334444 2021.9659297 URL mattps://dol. /rrg
10. 1109/ ICEET 2022 01022.9969697 9659697
Yujun Shen, Ceyuan Yang. Xiaoou Tang, and Bolei Zhou Interfacegan: Interpreting the disentan- gled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 2020.
Hengcan Shi, Munawar Hayat, and Jianfei Cai. Unpaired referring expression grounding via bidi- rectional cooss-modal matching. arXiv preprint vrXiv:2201.06688, 2022a.
Hengcan Shi, Munawar Hayat, Yicheng Wu, and Jianfei Cai. Proposalclip: Unsupervised open-
category object proposal generation via exploiting clip cues. arXiv preprint arXiv:2201.06696 2022b.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Yacl Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically object sketching. arXiv preprint arXiv:2202.0582. 2022.
Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12863 12872, 2021.
Weihao Xia, Yujuu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2256-2265, 2021.
Zipeng Xu, Tianwei Lin, Hao Tang, Fu Li, Dongliang He, Nicu Sebe, Radu Timofte, Luc Van Gool, and Errui Ding. Predict, prevent, and evaluate: Disentangled text-driven image manipulation empowered by pre-trained vision vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18229-18225 18238, 2022.
Ansheng You, Chenglin Zhou, Qixuan Zhang, and Lan Xu. Towards controllable and photorealistic region-wist image manipulation. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 535-543, 2021.
12

---

Under review as a conference paper at ICLR 2023
Yun Zhang, Ruixin Liu, Yifan Pan, Dehao Wu, Yuesheng Zhu, and Zhiqiang Bai. Gi-ace: Gan inversion based attentive expression embedding network for facial expression editing. 2021 IEEE International Conference Image Processing (ICIP), pp.
IEEE, 2021.
Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipu- lation the natural image manifold. In European conference on computer vision, pp. 597-613. Springer, 2016.
Alara Zindancroglu and Metin Sezgin. Perceptually validated precise local editing for facial action units with stylegan. arXiv preprint arXiv:2107.12144 2021.
A APPENDIX
A.1 SIMILARITY OF DIFFERENT CLIP EMBEDDINGS
As a complement to Figure we record the averaged cosine similarity of the embeddings from different modalities in Table We see that image embeddings and text embeddings lie in different regions in C and have lower inter-modality similarity compared to the intra-modality similarity, regardless of their semantic meanings. From the Table similarity to images (0.491 and 0.493) than texts to images (0.200).
also that PAEs have higher cosine
Table 2: Averaged cosine similarity of different CLIP embeddings. Note that image embeddings and text embeddings have lower inter-modality similarity. Also note that PAEs have higher intra- modality similarity (lower variation) than images, showing that they discard some noisy information.
face text
emotion text
hair text
face image
CPCA emotion
CPCA hairstyle
face text emotion text
0.852
0.862
0.779
0,200
0.684
0.630
0.877
0.784
0.198
0.693
0.633
hair text face image CPCA eemotiion
0.764
0,201
0.634
0.622
0.567
0,491
0.493
0.771
0.731
0.725
huirstyle
Note that this discrepancy of image embeddings and text embeddings in the CLIP space is general and not restricted faces. We plot the embeddings of 100 random images and 100 random texts visualized using PCA in Figure 6 and also record the similarity in Table 3 We note the same observation that image embeddings and embeddings lie in different regions in C and have lower inter-modality similarity compared to the intra-modality similarity.
Table Averaged cosine similarity of CLIP embeddings 100 random images and texts. Note that image embeddings and text embeddings have lower inter-modality similarity.
image
text
image text
0.664
0.199
0.845
A.2 MORE OPTIONS FOR PROJECTION AND AUGMENTATION
In this section we present two more options for projection and three more for augmentation. About notations: give each option a short identifier (or all) and add it to the superscript of Pay, Am,r Egg. For example, Egg onto orthonormal basis (PC5) and augmentation that preserves the coefficients (A
eSS means the cnojection-aummentee
embedding with projection
13

---

Under review as a conference paper at ICLR 2023
random texts
0.4
random images
0.2
0.0
-0.2
0.4
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
Figure 6: PCA visualization of 100 random images and 100 random texts in the CLIP space. Note that image and text embeddings lie in differet regions.
A.2.1 PROJECTION
A simpler type of projection, Pwn also assumes 20 to be linear subspace of the CLIP space E and very similar to PGS except that it does not orthonormalize the basis vectors.
Pan(e,) = Σ (b by.
(10)
Note that in this case, the dot product with the basis vectors is not strict projection onto linear subspace. However, this simple option also worked well in our prior experiments. This is possibly due to the high dimensionality of the CLIP space (512): if we do not have many basis vectors (e.g., < 20), their pairwise dot products tend to be very small and thus are nearly orthogonal to each other already.
The second option, PAAl. does not assume that 20 is necessarily a linear subspace. Instead, it can be any manifold. 'PAl directly stores the set T. By storing embeddings of all related texts, it is effectively sampling and storing the points on the manifold 20, and when performing the projection onto surface, the best approximation would be to pick the point on the surface that is closest to the vector be projected:
PAll (er)
arg min CosSim(er, e)-
(11)
eET
Naturally, if we obtain larger T, we sample more points from the manifold and better approximate the projection.
A.2.2 AUGMENTATION
We propose three more option for the augmentation operation: Agn T. Als T and A The first one is used with projections Pas, Pen and PPCA, and the last two with PAll
The simplest A adds the text and image together, with coefficient a controlling the augmenting power:
A (w))(())
aer-
(12)
However, simply adding up er may result vector too similar to the original embedding, likely resulting in an optimization process barely doing anything. In that case, the augmentation introduced equation 19 better option because it additionally weakens a certain amount on other components (while preserving the sum of coefficients). To avoid the misunderstanding, we denote by Agu, r this
simplest augmentation and denote by ATD,T the augmentation in equation 9
The last two types of augmentation, ABr and A are for PAll, Since PAll gives an embedding T. which is expected indicate the current feature of the image, if want to change that
14

---

Under review as a conference paper at ICLR 2023
feature to be the one specified by the text T, we can simply exchange with the text embedding exi AQT. (w)) = w + a (er- - -Pei (e1)).
(13)
where a is the augmenting power.
The final Abgh. is more robust version than Abe.r Instead of doing the one-for-one exchange as above, we can do one-for-a exchange. More precisely, we still add in a copies of er, but instead of subtracting a copies of the most similar text embeddings, we subtract the different, most similar text embeddings (each embedding only subtracted once).
Naturally, between the one-for-one exchange in Abi,r and one-for-a exchange A there exist many other options from one-for-k for any & between 1 and a. We leave this exploration future work.
A.3 EVALUATION CRITERIA
In the future other researchers may want to develop new types of PAE for their specific tasks. In this section we provide two straightforward criteria to evaluate and select different options of PAE:
1. We need CosSim(Emn e1,) be large possible, where e1, is the embedding of the target image, i.e. the ideal target. This is because our goal to approximate the inaccessible et.- At least, need CosSim et,) CosSim(er,, es,) that we gain something by replacing er with Egg as an optimization target.
2. We need CosSim (&g,, er,) CosSim(Enn er), otherwise PAE is similar the orig- inal image and the no change towards e1, will be made during the optimization.
We evaluate all the eight options of PAE (two proposed in Section 4.1 and six in Appendix A.2 in the text-guided facial emotion editing experiment described in Section PGS is realized by using six basic human emotions (Ekman T992) as semantic basis:
"happy", "sad", "angry", "fearful", "surprised". "dia""",""sgnunn "disgusted";
20 PPCA and Pay are implemented with a corpus T consisting of 277 emotion texts found online and
in Figure 7 Each color corresponds to an option of Egn. Restating the above criteria, within each color, we need 1. the solid line to be as large as possible and least be above the orange horizontal line; and 2. the dash-dotted line to be above the black horizontal line. We see that in this particular experiment of facial emotion editing, it is more suitable to use E with an a [5, 15). E with an E [10, 15), or ePCA,+ with an [2.5, 5]. Note that as the + versions perturb the embedding more, the dash-dotted lines for Egn> cPCA, are higher than their non-+ counterparts. and Egg
ten principal components. We plot the above quantities
A.4 A FAILED CASE: DOUBLE-PROJECEE EMBEDDING
This section presents the double projected embedding (DPE) that has similar idea of using CLIP subspaces extract relevant information from the embeddings. DPE is simpler than PAE in that it does not have the second step of augmenting the projected image embedding; instead, it projects both image and text embeddings onto the subspace and directly optimizes the image towards the text in the subspace.
We conduct the same text-guided emotion editing experiment as in Section 5 uusing DPEGS and DPEPCA with 20 principal components. The results are summarized in Figure and Figure 9 re- spectively.
We can see from the results that although most faces indeed changed the emotion according to the text prompts, we lost the disentanglement: in some manipulation, the lighting condition, the background, the hair, even the facial identity changed.
One possible explanation that by projecting faces onto the emotion subspace, most information except the emotion has been lost, so cannot preserve irrelevant attributes. Recall that in con- structing PAE, the final step is to add back the residual r to go back to the "image region" (see Eq. ). We hypothesize that this exactly contains the irrelevant information that need to pre- serve. This example also shows that the necessity and importance using an optimization target that is close enough to the embedding e, of the original image.
15

---

Under review as a conference paper at ICLR 2023
1.0
0.8
0.6
Similarity
0.4
sim , Ct.
0.2
0.0
-0.2
0.0
2.5
5.0
15.0 17.5 20.0 Augmenting power
7.5
10.0
12.5
s)
G: Sim(PAE eee))
M: Sim(PAEPCA .e.)
SimEPAE
sim(PAE.e)
H: Sim .e,)
N: sim(PAE .e,)
T: sim(PAEAI.
C: A-B
I: G-H
O: M-N
U: S-T
): sim(PAE* .
Sim(PAE + .e,,)
P: sim(PAEPCA, .e,))
V: sim(PAE
: simE * .e))
K: sim(PAEGS, ee)
Q: sim(PAERCA, e))
W: Sim(PAE
D-E
L:J-K
R: P-Q
X: V-W
Figure 7: Comparison of eight options of PAE a text-guided facial emotion editing experiment. Each option is color-coded. In order for Eny to be effective, need the solid line to be large as possible and at least be above the orange horizontal line and the dash-dotted line to be above the black horizontal line. In this experiment, the best three options are &gh (a E [5, 15), cGS,+ (a E [10, 15)) and Egg ePCA, (a 5)). Note also that the + versions perturb the embedding more, the dash-dotted lines for + versions are generally higher than their non-+ counterparts.
Orginal
Figure 8: Text-guided emotion editing using DPEGS Note that changes are not disentangled.
A.5 CLIP SUBSPACE EXTRACTS RELEVANT INFORMATION FROM TEXTS
In this section conduct two experiments to demonstrate that emotion subspace 20 can extract the relevant information from texts. The emotion subspace 21 created either using pGs or PPCA. but they result in similar observations.
16

---

Under review as a conference paper at ICLR 2023
Figure 9: Text-guided emotion editing using DPEPCA Note that changes are not disentangled.
A.5.1 EXTRACT EMOTION FROM EMOTION-ANIMAL PHRASES
We compare the averaged similarity of amotion-anima. phrases in the original CLIP space and in the emotion space 20 created by Pas. Each emotion-animal phrase consists of adjective for emotion (e.g., "happy", "sad") qualifying a noun for animal (e.g., "horse", "man"). 20 able to capture the emotion information, then expect that the similarity of phrases of same animal but different emotions (e.g., "happy horse", "sad horse") high in e (because they are the same animal) but is low in 2 (because the emotions different). On the contrary, if the emotion is the same but the animals are different (e.g., "happy horse", "happy cat"), then we expect that the similarity is high in 20 but low in
The result tabulated Table and coincide with our hypothesis: In 20, emotions dominate the similarity rather than the animals, showing that 20 can extract emotion information from the phrases.
Table 4: Averaged similarity of emotion-animal phrases. In 217, emotions dominate the similarity rather than the animals, showing that 2 can capture emotion information from the phrases.
same animal, different emotions same emotion, different animals
0.863
0.439
0.834
0.926
A.5.2 EXTRACT EMOTION FROM EMOTION TEXTS
We collect five emotions from each of the groups happy, sad, angry, disgusted, fearful (25 emotions in total) and use heat maps to visualize their pairwise similarity in C, in the emotion subspaces created by P and PPCA, The result shown Figure 10 We see that emotions in the same group have higher similarity in both emotion subspaces than in C.
A.6 FULL RESULT OF TEXT-GUIDED FACE EDITING
In this section we present the full result of the text-guided face editing that could not be placed in
Section to the space limit. Figures to 13 show the result of emotion, hairstyle and physical characteristic editing, respectively, using different PAEs: The text prompts as well as the type of PAE used are written to the left of each editing for different text prompts in the aforemmentioned three editing categories. Figure 27 shows the controllability of our method by varying the augmenting power.
Figure 14 to Figure 26 compare the eight models in face
17

---

Under review as a conference paper at ICLR 2023
187 sted
(a) In CLIP space
In P
digurad
dirated
(c) In PACA
Pairwise similarity of 25 emotions in five groups in three different spaces. We see that emotions in the same group have higher similarity in both emotion subspaces than in C.
Figure 10:
A.7 TEXT-GUIDED EDITING ON NON-FACIAL IMAGES
We also conduct text-guided editing on non-facial images, namely on images of dogs from AFHQ Dog dataset (Choi et al. 2020). The experiment procedure is very similar to Section 5 except for the three differences:
StyleGAN2 is pre-trained on the AFHQ-Dog dataset instead of the FFHQ dataset;
Since StyleCLIP PPatashnik al. 1021) and TediGAN (Xia al. 2021 do not release
StyleGAN2 model pre-trained on AFHQ-Dog, we only include the comparison of Naive, Naive+PAE, StyleMC, and StyleM++PAE;
The fur subspace is constructed by PPCA defined via extracting principal components from a corpus of nine fur color texts: white fur, black fur. orange fur, brown fur, gray fur, golden fur, yellow fur, red fur, and blue fur.
Figure 28 shows the aggregated result of Naive+PAE approach. The text prompts are written to the left of each row. Figure 29 to Figure compare the four models for different text prompts.
18

---

Under review as a conference paper at ICLR 2023
Original
NE
P-
SA0 FRE
Figure 11: Text-guided emotion editing using different PAEs
PAR
Figure 12: Text-guided hairstyle editing
19

---

Under review as a conference paper at ICLR 2023
Original
large eyes
PAE
small eyes FAE
small smallmmuse mouth large mouth FAE
PAE
Figure 13: Text-guided physical characteristic editing
Org
Navee
Noive+D0e
Styleiic StyfecL SWVECLIP+ORL IPPORE skyeecLiP
TedISAN-PPE TerIGNN StylOMC--PEE
Figure 14: Comparison of different models. The text prompt "happy".
20

---

Under review as a conference paper at ICLR 2023
Orgina
Nave
Nave-oit
StyVECLI-PRA StyjeCLIP
stylemic
Styleec -
TeaKSAN
Tec APA
Figure 15: Comparison of different models. The text prompt is "sad".
Original
Neive
Nave-PPE
styletic StyAECIPHHS PAME StyreCLIP
TeSGANN-RNEE TediSAN StyleMC- PALE
Figure 16: Comparison of different models. The text prompt is "angry""
21

---

Under review as a conference paper at ICLR 2023
Ongrnal
Nave
Nave+RREE
StyfeCUPPPIE StyrecUP
Styrenic
Sty Sty PHE
Tec CAN
Te01GAN--PE
Figure 17: Comparison of different models. The text prompt "surprised".
Orighnal
Natin
PeIGAN-NNI THE StyReC-- SIyAN: MyyICPPPRIC StyaCLIP NaNnerN
Figure 18: Comparison of different models. The text prompt is "blonde".
22

---

Under review as a conference paper at ICLR 2023
Cerohnll
Nalue
Sajeni Sky StyaCLIP-PI StyecUPP
THE Ted
Figure 19: Comparison of different models. The text prompt is "black hair".
Kales
20yeesi St Syartin-atee Sty ACUP
GNEGA TR-SOW 20AANC-hii
Figure 20: Comparison of different models. The text prompt is "grey hair".
23

---

Under review as a conference paper at ICLR 2023
Cerohnal
Nalee
Styaetir--I StyecUuP
SajeNi
THE Ted
Figure 21: Comparison of different models. The text prompt is "bald".
gha
hales
Seaetii-atie Sty ACUP
GAEGA-PP TR-SOW 20roNC-hiic S0yees
Figure 22: Comparison of different models. The text prompt is "curly hair".
24

---

Under review as a conference paper at ICLR 2023
Original
Nalve
Naive PALE
StyleCLPP
Stylakk StyleCLP P. PAE
StyleMC FARE
TeolDAN
TedIGAN PAE
Figure 23: Comparison of different models. The text prompt is "large eyes".
25

---

Under review as a conference paper at ICLR 2023
Original
Nalve
Nalve PALE
StyleCLP
Stylakk StyleCLI P. PAE
StyleMC FARE
TEOIDAN
TedIGAN PAE
Figure 24: Comparison of different models. The text prompt is "small eyes".
26

---

Under review as a conference paper at ICLR 2023
Original
Nalve
Nalve PALE
StyleCLPP
StylakE StyleCLP PAE
StyleMC FRRE
TEOlDAN
TedIGAN- PAE
Figure 25: Comparison of different models. The text prompt is "large mouth".
27

---

Under review as a conference paper at ICLR 2023
Oniginal
Naviv
Noive PPAE
StyleLIPP FPAE StyleCLIP
StyleMc
Style/C PPAE
TRoIGAN
TedlCAR FRRE
Figure 26: Comparison of different models. The text prompt is "small mouth".
Figure 27: Varying the augmenting power when making faces happy to show the controllability
28

---

Under review as a conference paper at ICLR 2023
Orginnl
black fur
whire tur
orarge fu
Figure 28: Text-guided physical fur color editing
Orignal
havee
NoIvE+RME
StyleNC
Styeli +AME
Figure 29: Comparison of different models. The text prompt is "black fur".
Orgginll
Nave
StyieHceWE styeeic
Figure 30: Comparison of different models. The text prompt is "white fur".
29

---

Under review as a conference paper at ICLR 2023
Onigial
Naive
Noive+9RE
Sy/Mic-Rt styleMc
Figure 31: Comparison of different models. The text prompt is "orange fur".
30


Reviews:
[{"review_id": "mn6kkXlRxz", "reviewer": "Reviewer_62cf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to achieve disentangled, interpretable and controllable text-guided image manipulation with CLIP-based models. The authors propose CLIP projection-augmentation embedding (PAE) to replace the original multimodal embedding as optimizing target. PAE is simple and can be easily to injected into other CLIP-based text-guided image manipulation models. Experimental results on face datasets shows that it can disentangle face attribute and even control the degree of attribute in the generated results. ", "summary_of_the_review": "Although I could recognize the simplicity and effectiveness of the method in single attribute facial editing, the method has limitation in more broader scenarios of text-guided image manipulation. ", "strength_and_weaknesses": "Strength:\n1.\tThis paper is well writing and easy-to-follow. \n2.\tThe analysis of embedding space well explains the motivation of the approach.\n3.\tThe proposed component (PAE) is reasonable and simple enough for re-implementation. It can be combined in other models.\n4.\tExperimental results show the effectiveness of the proposed PAE in text-guided face editing.\n\nWeakness:\n1.\tThe experiments are only conducted on face datasets which makes the title “text-guided image manipulation” over claimed.\n2.\tBesides Gram-Schmidt and dimension reduction techniques for project, clustering is also a possible solution. While the authors have not mentioned about other alternative methods.\n3.\tThe method seems limited in several scenarios:\na)\tNon-facial images, such as animals, architecture or other objects. \nb)\tTexts that are not attributes. It seems that attribute is better to be mapped into subspace while other texts may be difficult. \nc)\tAttributes with overlaps. There are some attributes that affect multiple parts in the face. For example, “black hair and wearing glasses”, “Asian woman”.\nd)\tAttributes that are not existed in training data. This also indicates the biasness of this method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "bxcBY7vOMKc", "reviewer": "Reviewer_Qecw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present an approach to edit images using text promp by projecting the image embedding to a subspace which is semantically meaningful to the concept being edited. The method integrates seamlessely into most popular CLIP based text driven editing approaches. State of the art quantitative results are shown for text based image editing on a dataset of faces.", "summary_of_the_review": "The authors provide a simple and elegant method for text based editing by projecting CLIP based embeddings  onto \"meaningful\" subspaces. However, the work would greatly benefit from better motivation of the projection and augmentation strategy. Furthermore, the results are only provided on a single class of data. Providing on few other object classes would potentially help aid the narrative for generalizability. ", "strength_and_weaknesses": "## Strengths\n\n1.  The authors propose a simple and elegant solution. The proposed approach is a general augmentation strategy that can be applied to a large class of text based editing approaches.   \n  \n2. The manuscript is well written with attention to detail. All the important components of the approach have been adequately explained\n\n3. Adequate treatment of the related literature has been provided to contextualize the proposed approach in relation to similar methods.\n\n3.  Code is provided to aid in reproducibility.\n\n5. The authors provide important insights regarding the discrete regions of space occupied by the text embedding and image embedding in the joint space. The solution accounts for this gap while influencing image space edits through text.\n\n6. The appendix provides much needed additional details regarding similarity of text and image embeddings and how to structure the projection and augmentation operation.\n\n## Weaknesses\n\n1. There are several simplifying assumptions made regarding (projection+augmentation) operation. Particularly,  \n \n&nbsp; &nbsp;&nbsp; &nbsp; a) Equation (2) and (3) assume that the all other attributes are orthogonal to the attributes captured by the basis of the corpus subspace. It is unclear how valid this assumption is in the general case. For instance, extending upon the examples demonstrated in the paper, mouth open->close and the emotion \"surprised\" may not truly be disentangled.  \n   \n&nbsp; &nbsp;&nbsp; &nbsp; b) The augmentation operation is akin to a linear operation in the subspace, again relying on the linearity assumption on the part of the space spanned by a particular attribute (say emotion). Is it not more likely that the space of emotion edits forms a nonlinear manifold, such that augmentation (or movement) in that subspace is not necessarily linear? There is some discussion about this in the appendix section, but a more detailed explanation of why the linearity assumption holds would be helpful.  \n\n&nbsp; &nbsp;&nbsp; &nbsp; c) The motivation for the augmentation step is unclear. Particularly, if $w$ is a projection of $e_I$ onto a subspace where all other attributes are disentangled why do we need to explicitly \"weaken\" components of $w$. \n\n&nbsp; &nbsp;&nbsp; &nbsp; d) A clearer explanation of what exactly equation (8) and (9) are doing would be insightful . Particularly, it seems like the $l_1$ norm of the $e_1$ in prompt-space is being shrunk and then a scaled version of $P(e_T)$ is being added back. Explanation regarding the function of this scaling term (or reference to works which employ a similar strategy to move in subspaces) would greatly improve readability of that section.  \n \n2. Do we need a different projection operation for every attribute to be controlled?\n4. Experiments are demonstrated on a limited dataset. The narrative would greatly benefit from showing more examples on a wider variety of datasets like (CUB/ Flowers), as it would highlight the generalizability of the approach.  \n\n&nbsp; &nbsp;&nbsp; &nbsp;\ta) Is the limited dataset an issue of annotated data?  \n\n&nbsp; &nbsp;&nbsp; &nbsp;  b) Are the limitation due to lack of pre-trained models for other datasets?\n\n5. A number of references to \"optimization\"  has been made. What exactly is the optimization process involved ? Since all the steps involved in both the projection and augmentation operators are deterministic and the pre-trained models being used are deterministic, what are the learnable/ optimizable parameters in the system.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "R2YTJ-QRZ5", "reviewer": "Reviewer_WWDd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed CLIP-PAE, which projects and augments CLIP image embedding and text embedding in different subspaces, and uses embedding in subspace to optimize the cosine similarity for text-guided image manipulation. It's applied to different baseline methods and outperforms them in both quantitative and qualitative results.", "summary_of_the_review": "Although some details about methodology and experiments, and limitations are not illustrated, the novelty, intriguing analysis, and strong performance make the paper above the bar from my point of view. ", "strength_and_weaknesses": "Strengths:\n1. I really appreciate the analysis in Section3. It contains intuitive visualization and a preliminary pilot study which provide an experimental view of why the original CLIP score doesn't work well in guiding image generation. \n2. The methodology, PAE, is novel. It subtly utilizes the relevant semantics in text to build basic vector subspace.\n3. The proposed subspace embedding can be generally used in many previous works that employ CLIP similarity for text-guided image manipulation. Also, the experimental results show that adding the proposed embedding can indeed improve the generation quality. \n\nWeaknesses:\n1. My concern in terms of methodology is that I still cannot get why the disentanglement is guaranteed in the proposed method, neither from theoretical proof nor intuitive understanding. If you have a look at Fig3 b, you can find the similarity score in the hairstyle subspace somehow also somehow increases. From the methodology, it does build the basic vectors by embeddings in the separate semantics but there might be some spurious/bias between selected semantics and other unwanted semantics, which is not specifically avoided.\n2. It's easy to find the relevant corpus of emotion or haircut subspace. But not all subspaces contain a clear definition of the relevant corpus. For example, for `Donald Trump`, as an example in StyleCLIP, it's hard to find a comprehensive and precise relevant corpus. \n3. The quality of the corpus matters for the proposed methods. However, it seems the corpus is manually added. There is no clear indicator for a good set of corpus until it's tried in the experiment and takes back the performance. Such a limitation hurts the generalization ability of the proposed method.\n4. In experiments:\n- From Fig5a, we can find with PAE, the unwanted semantics also change sometimes. For example, the color of the boy's hoodie changed to blue in StyleCLIP+PAE. So echoing what I mentioned before, entanglement seems still exist. It's not guaranteed that the samples are not cherry-picked. I wonder if there is any way we can quantitatively measure the disentanglement degree?\n- In Table 1, it's not in which \\alpha, the numbers are obtained. For the different metrics of same method, is the same \\alpha used?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C_a7pT0QrU", "reviewer": "Reviewer_DXMN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores the limitation of using CLIP to build a joint space for text and image, where the disentanglement, interpretability, and controllability are hard to guarantee in text-guided image manipulation. To mitigate this problem, a PAE is proposed, which defines corpus subspaces spanned by relevant prompts to capture specific image characteristics. Experiments of adopting the proposed PAE in current CLIP-based method improve their performance qualitatively and quantitatively.", "summary_of_the_review": "See above weaknesses. I am happy to raise my rating based on authors' responses.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well written and easy to follow.\n2. The proposed method can be adopted in most CLIP-based methods.\n\n\nWeaknesses:\n1. Authors only evaluate the limitation of CLIP on a face dataset, which is limited and might not be applied for a general dataset. \n2. Figure 1 only shows the distance between image features and original text embeddings, and image features and PAE text embeddings, respectively, which might not be enough to support the proposed problem of CLIP, because (1) it is a 2D visualization using PCA, which might not fully show the real distance in a high-dimensional space (2) the distance between them is hard to quantify the differences, without any reference, e.g., comparison with images from another categories.\n3. I am confused about why the proposed method allows a better disentanglement, and how to build the subspace? Do authors need to build multiple subspaces for different attributes, e.g., hair and expression?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject