Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper explores the limitation of using CLIP to build a joint space for text and image, where the disentanglement, interpretability, and controllability are hard to guarantee in text-guided image manipulation. To mitigate this problem, a PAE is proposed, which defines corpus subspaces spanned by relevant prompts to capture specific image characteristics. Experiments of adopting the proposed PAE in current CLIP-based method improve their performance qualitatively and quantitatively.
strength_and_weaknesses: Strength:

1. The paper is well written and easy to follow.
2. The proposed method can be adopted in most CLIP-based methods.


Weaknesses:
1. Authors only evaluate the limitation of CLIP on a face dataset, which is limited and might not be applied for a general dataset. 
2. Figure 1 only shows the distance between image features and original text embeddings, and image features and PAE text embeddings, respectively, which might not be enough to support the proposed problem of CLIP, because (1) it is a 2D visualization using PCA, which might not fully show the real distance in a high-dimensional space (2) the distance between them is hard to quantify the differences, without any reference, e.g., comparison with images from another categories.
3. I am confused about why the proposed method allows a better disentanglement, and how to build the subspace? Do authors need to build multiple subspaces for different attributes, e.g., hair and expression?
clarity,_quality,_novelty_and_reproducibility: The paper might lack sufficient details about the proposed method, which causes difficulties to reproduce the results.
summary_of_the_review: See above weaknesses. I am happy to raise my rating based on authors' responses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed CLIP-PAE, which projects and augments CLIP image embedding and text embedding in different subspaces, and uses embedding in subspace to optimize the cosine similarity for text-guided image manipulation. It's applied to different baseline methods and outperforms them in both quantitative and qualitative results.
strength_and_weaknesses: Strengths:
1. I really appreciate the analysis in Section3. It contains intuitive visualization and a preliminary pilot study which provide an experimental view of why the original CLIP score doesn't work well in guiding image generation. 
2. The methodology, PAE, is novel. It subtly utilizes the relevant semantics in text to build basic vector subspace.
3. The proposed subspace embedding can be generally used in many previous works that employ CLIP similarity for text-guided image manipulation. Also, the experimental results show that adding the proposed embedding can indeed improve the generation quality. 

Weaknesses:
1. My concern in terms of methodology is that I still cannot get why the disentanglement is guaranteed in the proposed method, neither from theoretical proof nor intuitive understanding. If you have a look at Fig3 b, you can find the similarity score in the hairstyle subspace somehow also somehow increases. From the methodology, it does build the basic vectors by embeddings in the separate semantics but there might be some spurious/bias between selected semantics and other unwanted semantics, which is not specifically avoided.
2. It's easy to find the relevant corpus of emotion or haircut subspace. But not all subspaces contain a clear definition of the relevant corpus. For example, for `Donald Trump`, as an example in StyleCLIP, it's hard to find a comprehensive and precise relevant corpus. 
3. The quality of the corpus matters for the proposed methods. However, it seems the corpus is manually added. There is no clear indicator for a good set of corpus until it's tried in the experiment and takes back the performance. Such a limitation hurts the generalization ability of the proposed method.
4. In experiments:
- From Fig5a, we can find with PAE, the unwanted semantics also change sometimes. For example, the color of the boy's hoodie changed to blue in StyleCLIP+PAE. So echoing what I mentioned before, entanglement seems still exist. It's not guaranteed that the samples are not cherry-picked. I wonder if there is any way we can quantitatively measure the disentanglement degree?
- In Table 1, it's not in which \alpha, the numbers are obtained. For the different metrics of same method, is the same \alpha used?

clarity,_quality,_novelty_and_reproducibility: The writing is clear and different variations of the proposed method are discussed. Quality and originality are good.
summary_of_the_review: Although some details about methodology and experiments, and limitations are not illustrated, the novelty, intriguing analysis, and strong performance make the paper above the bar from my point of view.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present an approach to edit images using text promp by projecting the image embedding to a subspace which is semantically meaningful to the concept being edited. The method integrates seamlessely into most popular CLIP based text driven editing approaches. State of the art quantitative results are shown for text based image editing on a dataset of faces.
strength_and_weaknesses: ## Strengths

1.  The authors propose a simple and elegant solution. The proposed approach is a general augmentation strategy that can be applied to a large class of text based editing approaches.   
  
2. The manuscript is well written with attention to detail. All the important components of the approach have been adequately explained

3. Adequate treatment of the related literature has been provided to contextualize the proposed approach in relation to similar methods.

3.  Code is provided to aid in reproducibility.

5. The authors provide important insights regarding the discrete regions of space occupied by the text embedding and image embedding in the joint space. The solution accounts for this gap while influencing image space edits through text.

6. The appendix provides much needed additional details regarding similarity of text and image embeddings and how to structure the projection and augmentation operation.

## Weaknesses

1. There are several simplifying assumptions made regarding (projection+augmentation) operation. Particularly,  
 
&nbsp; &nbsp;&nbsp; &nbsp; a) Equation (2) and (3) assume that the all other attributes are orthogonal to the attributes captured by the basis of the corpus subspace. It is unclear how valid this assumption is in the general case. For instance, extending upon the examples demonstrated in the paper, mouth open->close and the emotion "surprised" may not truly be disentangled.  
   
&nbsp; &nbsp;&nbsp; &nbsp; b) The augmentation operation is akin to a linear operation in the subspace, again relying on the linearity assumption on the part of the space spanned by a particular attribute (say emotion). Is it not more likely that the space of emotion edits forms a nonlinear manifold, such that augmentation (or movement) in that subspace is not necessarily linear? There is some discussion about this in the appendix section, but a more detailed explanation of why the linearity assumption holds would be helpful.  

&nbsp; &nbsp;&nbsp; &nbsp; c) The motivation for the augmentation step is unclear. Particularly, if $w$ is a projection of $e_I$ onto a subspace where all other attributes are disentangled why do we need to explicitly "weaken" components of $w$. 

&nbsp; &nbsp;&nbsp; &nbsp; d) A clearer explanation of what exactly equation (8) and (9) are doing would be insightful . Particularly, it seems like the $l_1$ norm of the $e_1$ in prompt-space is being shrunk and then a scaled version of $P(e_T)$ is being added back. Explanation regarding the function of this scaling term (or reference to works which employ a similar strategy to move in subspaces) would greatly improve readability of that section.  
 
2. Do we need a different projection operation for every attribute to be controlled?
4. Experiments are demonstrated on a limited dataset. The narrative would greatly benefit from showing more examples on a wider variety of datasets like (CUB/ Flowers), as it would highlight the generalizability of the approach.  

&nbsp; &nbsp;&nbsp; &nbsp;	a) Is the limited dataset an issue of annotated data?  

&nbsp; &nbsp;&nbsp; &nbsp;  b) Are the limitation due to lack of pre-trained models for other datasets?

5. A number of references to "optimization"  has been made. What exactly is the optimization process involved ? Since all the steps involved in both the projection and augmentation operators are deterministic and the pre-trained models being used are deterministic, what are the learnable/ optimizable parameters in the system.
clarity,_quality,_novelty_and_reproducibility: *Clarity:* The authors present their approach in a detailed manner and the narrative is meaningful. Although some parts of the approach section can be made clearer as indicated in the section above. Some of the design decisions made for the augmentation is unclear. 

*Quality:* Although authors demonstrate improvements quantitatively, the qualitative results demonstrated sometimes don't seem to show much change for certain attributes. The quality of writing, however, is informative and engaging.

*Novelty:* The ideas themselves might not be entirely novel since they are leveraging basic ideas from linear algebra, but the insights provided are potentially helpful for textual editing of most latent space based generative models.

*Reproducibility:* The equations, architecture and data are explained clear and aids in reproducibility. The code  is also provided to help reproduce the metrics reported in the manuscript. 
summary_of_the_review: The authors provide a simple and elegant method for text based editing by projecting CLIP based embeddings  onto "meaningful" subspaces. However, the work would greatly benefit from better motivation of the projection and augmentation strategy. Furthermore, the results are only provided on a single class of data. Providing on few other object classes would potentially help aid the narrative for generalizability.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to achieve disentangled, interpretable and controllable text-guided image manipulation with CLIP-based models. The authors propose CLIP projection-augmentation embedding (PAE) to replace the original multimodal embedding as optimizing target. PAE is simple and can be easily to injected into other CLIP-based text-guided image manipulation models. Experimental results on face datasets shows that it can disentangle face attribute and even control the degree of attribute in the generated results. 
strength_and_weaknesses: Strength:
1.	This paper is well writing and easy-to-follow. 
2.	The analysis of embedding space well explains the motivation of the approach.
3.	The proposed component (PAE) is reasonable and simple enough for re-implementation. It can be combined in other models.
4.	Experimental results show the effectiveness of the proposed PAE in text-guided face editing.

Weakness:
1.	The experiments are only conducted on face datasets which makes the title “text-guided image manipulation” over claimed.
2.	Besides Gram-Schmidt and dimension reduction techniques for project, clustering is also a possible solution. While the authors have not mentioned about other alternative methods.
3.	The method seems limited in several scenarios:
a)	Non-facial images, such as animals, architecture or other objects. 
b)	Texts that are not attributes. It seems that attribute is better to be mapped into subspace while other texts may be difficult. 
c)	Attributes with overlaps. There are some attributes that affect multiple parts in the face. For example, “black hair and wearing glasses”, “Asian woman”.
d)	Attributes that are not existed in training data. This also indicates the biasness of this method.

clarity,_quality,_novelty_and_reproducibility: This paper is well written and easy to follow. The proposed method is reasonable and novel in limited scenarios. Considering the simplicity of the method, the reproducibility should be high.

summary_of_the_review: Although I could recognize the simplicity and effectiveness of the method in single attribute facial editing, the method has limitation in more broader scenarios of text-guided image manipulation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed a new approach with CLIP to enforce better disentanglement, interpretability and controllability for text-guided image manipulation. Reviewers have raised major concerns about weak technical contributions (weak novelty and lack of strong theory justification) and weak empirical studies (e.g., the datasets are restricted only to face while the paper is claimed to address general image domains, lack of comprehensive ablation studies to justify why and when it works or not), etc. Authors have tried to address some of review questions in their rebuttal, however, the work may need more a thorough revision and improvement before it can be ready for acceptance.
Decision: Reject