Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new method for semantic image editing, which allows users to edit images by changing their segmentation masks. More specifically, the method embeds an image required to be modified into the GAN's latent space and then perform optimisation according to the segmentation edit.
main_review: Originality: The proposed method builds on DatasetGAN and SemanticGAN, but allows users to achieve image editing. The difference between proposed method and previous works have been shown in related work.

Quality: The submission is technically sound, claims are well supported, and authors have discussed strengths and weaknesses of the proposed method.

Clarity: The paper is well organised and easily followed.

Significance: The results look good, which achieve high-quality image editing. 


limitations_and_societal_impact: Limitations: 
1. The proposed editing method relies heavily on the shared latent code between images and their segmentation masks. However, image and segmentation masks are different modal information, so how do authors ensure these two features are exactly match in latent space?

2. The method depends on the quality of segmentation masks, so if the generated masks fail to separate each part of an object, can proposed method still modify desired regions? e.g., in cat and bird examples, the proposed method seems to have limited modification choices.

3. Can proposed method control the scale of modified effect?

4. The proposed method builds on top of DatasetGAN, relying on modifying generated segmentation masks. Although authors show some differences in line 102, it might be better to discuss more about their differences or highlight their contribution.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work proposes EditGAN for semantic image editing with pre-trained GANs. It first introduces an additional branch beyond the original GAN generator to predict the semantic map corresponding to the image synthesis (this part is proposed by previous work). It then learns an encoder to project a target image to a latent code (this part is also proposed by previous work). At this end, given an image with the associated latent code and segmentation map, EditGAN enables semantic image editing by (1) modulating the segmentation map, (2) optimizing the latent code to fulfill the segmentation change, and (3) defining the semantic vector as the difference between the initial latent code and the optimized code. It turns out that such semantic vector is applicable to other images.

main_review: Pros

- Good editing results, especially for the high-precision editing on details, such as spoke-rotation in Fig. 7.


Cons

- Limited novelty. The generator (StyleGAN2), segmentation branch (DatasetGAN), GAN inversion (pSp) are all proposed by previous work. Editing the segmentation map is able to edit the image correspondingly is not that surprising. From this perspective, the technical novelty is incremental.

- Lack of an important baseline. If I understand correctly, in the process of learning the edit vectors, users are required to modulate the segmentation map to make the modulation reasonable. For example, in Fig. 1, enlarging the wheels yet keeping the segmentation map reasonable is not that trivial. Then, a straightforward pipeline would be "what if directly changing the image, like using PhotoShop?". From my point of view, editing the image has the same difficulty as editing the segmentation map. In this way, we may not need the segmentation branch. More concretely, given an image A and an edited (with PhotoShop) image B, we can invert both of them to the latent space and get two latent codes, a and b. Then, (b-a) defines a semantic vector. According to my experience, such an operation is able to achieve wearing eyeglasses on face models. Without the comparison to such a baseline, I am not convinced that the proposed approach can give better results, but it requires much more effort than the aforementioned baseline.

- Limited editing flexibility. 
(1) Is it possible to change the appearance of a particular segmentation part, e.g., changing the hair color? 
(2) Is it possible to make the person one eye looking left and the other right?
(3) Enlarging ear and deleting ear in Fig. 4 do not make sense to me. Also, the ears are not fully deleted. 
(4) When shrinking a certain part, how to fill the missing pixels seems to be a problem. Taking the first car on the 3rd row of Fig. 5 as an example, the missing parts are blurry after shrinking the wheels.

limitations_and_societal_impact: The paper mentioned the broader impact in Sec.6.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The authors proposed a local image-editing algorithm based on pre-trained StyleGAN2. The motivation is simple. Editing is performed based on two branches: generation and parsing. Parsing relies on pre-defined segmentation masks. The incremental editing vector is learnt by constraints to edited images and associated parsing masks. To handle the disentanglement, additional optimization is needed to perform refinement. 
main_review: The paper is based on previous works: DatasetGAN and SemanticGAN. It can be viewed as a specific application of these two works on part editing.  But the improvement of the algorithm for image editing is nearly trivial. Besides, the similar idea is also applied in the following paper

Paint by word
https://arxiv.org/pdf/2103.10951.pdf

Replacing the NLP part with semantic masks in the above paper, the remaining core idea of solving editing vectors is nearly the same. Compared with Image2StyleGAN and Image2StyleGAN++, the improvement is the parsing branch, which is the idea of DatasetGAN and SemanticGAN. So, the novelty of the idea is too trivial compared with existing methods. I don't think the current state of the paper is suitable for NeurIPS publication. 


%%%%%% comments after rebuttal %%%%%%

Thank you for your detailed feedback. I agree with the authors that they proposed a simple method for local image editing. But the limited novelty is obvious. I still think it is obviously below the bar of NeurIPS acceptance. We need to encourage innovation. You may revise your paper according to Reviewers' concerns, especially Reviewer M7u3 who gave constructive advice from several different aspects.  
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposed a semantic image editing method that allows users to edit an input image by manually edit the corresponding parts of its predicted semantic segmentation map.
main_review: Overall, I think this paper is incremental and does not have enough contributions to be published at top venues:
1. This paper uses a combination of well-known methods: i) the joint modeling of images and their semantic segmentation [2][3] and ii) GAN inversion [4].
2. This is not the first paper that enables high-precision semantic image editing, e.g. SEAN [1]. In addition, methods like SEAN are more flexible as they enable users to control the styles in each semantic region while the proposed method cannot.
3. Compared to SEAN or similar methods, the major merit of the proposed method is that it uses less labeled data. However, this is a feature of previous studies [2][3].

So I think it is okay but not good enough.

[1] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region adaptive normalization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.

[2] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. arXiv preprint arXiv:2104.06490, 2021. 

[3] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. arXiv preprint arXiv:2104.05833, 2021.

[4] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE International Conference on Computer Vision, pages 4432–4441, 2019.

**Originality**: This paper tackles an existing task using a new combination of existing methods (i.e. joint modeling of images and their semantic segmentation [2][3], GAN inversion [4]). As abovementioned, the contributions are incremental.

**Quality**: I think the methodology and claims of this paper are convincing as most of them are similar to those in [2][3] and [4]. The major problem is that it lacks new insights and is incremental.

**Clarity**: The submission is clearly written and easy to follow. 

**Significance**: I think the results are okay but not good enough. As a combination of joint modeling of images and their semantic segmentation [2][3] and GAN inversion [4], the results are not surprising at all.

limitations_and_societal_impact: Yes, the authors discussed both limitations and societal impact.
However, there is one missing limitation: the proposed method can only control the shape of the semantic editing regions but not their contents.
review_title: null

Meta-review: The paper proposes a new method for GAN-based image editing with the pipeline where a user modifies a segmentation mask of an image. While the paper is well-written and the editing results are impressive, four reviewers unanimously consider the technical novelty of the paper as limited and the paper being incremental. Since none of the reviewers advocated for this work, my recommendation is "Reject".
Decision: Accept (Poster)