Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new architecture for using CLIP to modify images in coordination with StyleGAN. The system is known as FF-CLIP (Freeform CLIP) and incorporates semantic alignment and injection blocks to identify a semantic subspace in the GAN embedding space. Cross-attention is employed to achieve semantic alignment, while an injection technique previously used in HairCLIP is employed to inject the text embedding semantics into GAN space. The authors provide visual examples of image transformations and provide quantitative evaluations by human subjects indicating that humans prefer FF-CLIP modified images.
strengths_and_weaknesses: Originality
Strengths: The authors choose a relatively novel area of inquiry, that of providing text based guidance to a synthetic image generator, in this case a GAN. This is an area of much innovation at the moment.
Weaknesses: This seems very similar to other recent CLIP-based image modification systems, especially StyleCLIP. While the authors report outperformance of StyleCLIP in terms of human evaluation, it is difficult for this reader to tell that much of a difference between the StyleCLIP generated images and the FF-CLIP generated images based on the examples provided. The semantic injection technique has itself been used previously in HairCLIP.

Quality
Strengths: It is an interesting paper and to my knowledge the alignment/injection architecture is novel and might be useful.
Weaknesses: Evaluations are not robust enough to be convincing to this reader: the authors evaluate only six image modifications based on human preferences. It’s hard to know whether the results are significantly better than those achieved by, e.g., StyleCLIP when the evaluation is narrow.

Clarity
Strengths: Good visualizations, and the math is presented in a straightforward and comprehensible manner.
Weaknesses: The writing can be hard to follow, with grammatical issues throughout. 

Significance
Strengths: Synthetic image generation is gaining ground rapidly and this system may help to advance the state of the art.
Weaknesses: The proposed system mainly consists of components previously used in very similar systems such as StyleCLIP and HairCLIP. If the evaluation of the system were more robust, this might not be an issue, but it is also hard to gauge how much of an advance this model is.
questions: Does model size of the CLIP used make any difference? The larger CLIPs do much better on evaluations like ImageNet, so maybe there would be some benefit to using them here?
limitations: CLIP is known to encode humanlike biases (see Wolfe et al. 2022, ACM FAccT; Birhane et al. 2021, arXiv). It would be good to mention these in a limitations section. How might they affect the system being developed?
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, an image manipulation method FFCLIP is proposed to edit image semantics based on the text prompt guidance. Following the StyleGAN latent space W for image generation, the alignment between semantics in the latent space W and text encoder is crucial for text-driven image editing. A semantic modulation method is developed to align the visual representations and text semantic embeddings via linear transformation where the parameters are computed based on cross attention computations. This effective alignment improves the previous empirical alignment design so that text prompts with different semantics can be taken for image editing via one model, while the empirical alignment confines each text prompt corresponds to one model. Experiments show that for each dataset (portraits, cars, or churches images), using one model is able to free-formly edit images based on different text prompts. The supplementary videos have also shown the effectiveness.
strengths_and_weaknesses: + The alignment is effective to automatically correspond text prompt embeddings to StyleGAN latent Space. This enables different semantical embeddings to be reflected in the visual image representations for manipulations. 
+ The semantic modulation first aligns text-image semantics via semantic alignment and then injects text prompts semantic. This design is reasonable to modify StyleGAN latent space since this space is not disentangled completely at present. Without latent space disentanglement, using the alignment via cross attention computation matches the text-image semantics. 
+ Experiments on three benchmark datasets show that under each dataset, using one model is able to edit image semantics free-form based on different text prompts with single or multiple semantics. The visual results are impressive on both paper and the supplementary video. The numerical results also show the effectiveness of FFCLIP.

- The existing methods TediGAN, StyleCLIP, HairCLIP also edit images based on the text information. A more in-depth analysis shall be taken to enhance the proposed contributions. For instance, TediGAN seems to tackle image editing with different text attributes. StyleCLIP uses one model for a specific text-image editing. HairCLIP focus on hair and can handle different text semantics. These would strengthen why the proposed FFCLIP makes senses and contribute to the cross-modal image manipulation area.
- A discussion with diffusion models would be helpful to emphasize the importance of editing StyleGAN latent space to benefit future research.

questions: See weakness. A clear discussion upon previous text-driven methods and more recent diffusion models would strengthen the proposed method.
limitations: As illustrated in Ln 271, the disentanglement of styleGAN latent space is an open problem. The negative society impact is for face malicious, which can be mitigated via deepfake detection methods.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an attention-based mechanism for improving CLIP- based StyleGAN editing models. 

Specifically, the paper takes the ‘latent mapper’ approach, where a neural network is trained to predict a latent-code change that would modify specific image attributes described through text. It argues that existing methods are trained to tackle specific texts, and therefore do not need to be aligned with CLIP’s space, limiting their ability to tackle novel texts.

The authors propose to better align the spaces through an attention mechanism, where the CLIP-text embedding is used to predict both layer and channel importance for latent code changes. This approach allows the authors to train their mapper on a larger collection of texts, creating better alignment between the CLIP and StyleGAN spaces, and enabling more disentangled modifications, with some generalization to unseen texts (e.g. combinations of texts seen during training).
strengths_and_weaknesses: Strengths:
1) The paper introduces a conceptually simple yet well motivated approach for identifying the StyleGAN layers and channels most relevant to the change of specific attributes described through text – specifically, attention.  
2) While there are prior works which deal with attention mechanisms in the context of StyleGAN inversion and editing [1, 2], they are, so far as I am aware, unpublished. However, I would urge the authors to cite these as related work (and consider comparisons to [2], which tackles the same task).
3) The results are generally of superior quality when compared to existing approaches. In particular, they are more disentangled, and allow for more complex manipulations than the baselines.

Weaknesses:

1) Clarity: The paper itself is very difficult to follow. I had a very hard time understanding the abstract and the introduction, and had to delve into the method and the experiments themselves to get any idea of what the authors are trying to do. The paper is not so complex that this should be required. I urge the authors to, at the very least, run their paper through Grammarly or similar tools.

2) Justification or experimental verification of claims: The core paper contains many claims which are unsubstantiated. In many cases, the supplementary **does** contain experiments which, at least in part, provide backings for these claims. However, they are never referenced from the core text, and if I did not read the appendix I would not be aware of them. These should absolutely be included in the main paper. A few examples:

    * The authors claim time and again that their model aligns the CLIP and StyleGAN spaces. The only experiment which possibly hints at this is the interpolation experiment in the appendix. Other than that, it appears that all experiments use the same, or compositions of the texts used during training. 

    * It is unclear from the paper itself whether the benefit of their approach is a result of the attention mechanism (i.e. improved latent mapper architecture), or the fact that they train for multiple prompts. There is one experiment in the appendix that gives this some backing (Figure 7, the w/o S&T experiment). However, it would also be interesting to see that the original latent mapper can’t be trained conditionally on multiple prompts.


3) Some (minor) wrong claims:
   * In the paragraph of L107, the authors suggest that inversion is done into W+ because it is better for editing. This is not the case. Inversion is done into W+ because it allows for better identity preservation. W+ inversions typically behave worse under editing than their W counterparts. See for example e4e [3].
   * The authors claim that StyleCLIP needs to train a different model for each textual prompt. This is only true for their latent mapper approach, but is not actually specified anywhere in the current submission. The ‘Global Directions’ method, for example, does not suffer from this limit. Actually, I would clarify early in the paper that this method specifically aims to improve on the latent mapper approach of StyleCLIP (this also ties to clarity above).

[1] Transforming the Latent Space of StyleGAN for Real Face Editing, Li et al, 2021

[2] FEAT: Face Editing with Attention, Hou et al, 2022

[3] Designing an Encoder for StyleGAN Image Manipulation, Tov et al, SIGGRAPH 2021

questions: 1) How well does the model generalize to unseen texts? Can a model not trained on hair-related prompts be used to change hair color? What if the only hair-related prompt was ‘blond hair’, would it still work to create red hair? What if the texts contained only ‘blond hair’ but also ‘red beard’, would compositionality in this manner work? If the model truly aligned the CLIP and StyleGAN spaces, one would expect this to be possible.

2) I’m a bit surprised that StyleCLIP fails so badly on hair color changes. Their official models have a purple hair mapper which works fine, and I just tried the global directions in their notebook and was able to generate ginger hair. Is there any chance that you trained their mapper with the fine-layers disabled? 

limitations: The limitations and social impact section of the paper is extremely bare-bones, consisting of 2.5 lines. 

A few likely limitations might be things discussed in my questions above: How well does the model generalize beyond the training text? 
Another limitation might be the fact that you train specifically on e4e codes. What if better encoders come along and now, I want to use those? Can you generalize to latent codes derived through optimization? Using another encoder? 

On the matter of social impact, this type of model can lead to harm in more than just the ability to create fake imagery. It can also propagate biases or have negative effects on minority representations. For example, in your Fig 6, the top-right individual’s skin color changes drastically after editing. It is fine that your model has these shortcomings, but the purpose of the limitations / impact paragraph is to be upfront about them.
review_title: null

Meta-review: metareview: The paper develops an image manipulation method FF-CLIP (Freeform CLIP) to edit image semantics based on the text prompt guidance. A cross-attention module is developed to align the visual representations and text semantic embeddings. The results show the effectiveness of the approach. Reviewers had concerns on the novelty and comparison with previous similar image editing methods. Yet the reviewer-author discussion effectively addressed some of the concerns, and two reviewers raised their scores. The ethics reviewers expressed concerns on the potential ethics issues as one of the major applications of the work is human face editing, which is known to amplify biases. Though the revised version has added more discussion on the ethnics issue, more in-depth discussion/analysis on potential solutions would be desired. 
confidence: Certain
Decision: Accept