EditGAN: High-Precision Semantic Image Editing

Abstract: Generative adversarial networks (GANs) have recently found applications in image editing. However, most GAN-based image editing methods often require large-scale datasets with semantic segmentation annotations for training, only provide high-level control, or merely interpolate between different images. Here, we propose EditGAN, a novel method for high-quality, high-precision semantic image editing, allowing users to edit images by modifying their highly detailed part segmentation masks, e.g., drawing a new mask for the headlight of a car. EditGAN builds on a GAN framework that jointly models images and their semantic segmentation, requiring only a handful of labeled examples – making it a scalable tool for editing. Specifically, we embed an image into the GAN’s latent space and perform conditional latent code optimization according to the segmentation edit, which effectively also modifies the image. To amortize optimization, we find “editing vectors” in latent space that realize the edits. The framework allows us to learn an arbitrary number of editing vectors, which can then be directly applied on other images at interactive rates. We experimentally show that EditGAN can manipulate images with an unprecedented level of detail and freedom while preserving full image quality. We can also easily combine multiple edits and perform plausible edits beyond EditGAN’s training data. We demonstrate EditGAN on a wide variety of image types and quantitatively outperform several previous editing methods on standard editing benchmark tasks.


Reviews and author responses:

The authors propose a novel approach to identify meaningful and well-localized images transformations based on a generative model that modulates a joint distribution of images with segmentation masks. The results look promising though there are concerns about the experimental section and baselines.  The approach proposed in the paper is based on recent works [1, 2]. That works modulate a joint images-segmentation distribution $p(x, y)$ in a few-shot regime with a common latent space for images and segmentations. Given a generated samples pair of an image and a mask $X, Y = G(w)$ and a hand-crafted target transformed mask $Y_{\mathrm{edited}}$, the authors propose to identify a meaningful latent deformation $\delta w^+$ by optimizing an objective that pushes $G^Y(w + \delta w^+)$ closer to $Y_{\mathrm{edited}}$ while preserving pixels of an untouched domain of the generated image $G^X(w + \delta w^+)$. This approach is new and seems to have high image editing quality. Nevertheless, I have a number of concerns about the evaluation and content that I describe below.

Strength:
- The paper proposes a novel approach for a few-shot GAN images manipulation. This approach looks quite simple and elegant and demonstrates good qualitative results.

Weakness:
- One of the fundamental limitations of the proposed method is that it is applicable only to manipulations that rely on image segmentation. I can see no way, how it can unroll light direction, luminance, background removal, texture and color change, age, and many other latent directions, reported in the prior works.
- It is not clear, if the method may modify pixels of the same semantic class independently. For instance, can it magnify only the front wheels of a car, remaining the back wheels unchanged?
- lines 206-213: once the two-stage scheme with refinement optimizes loss (5), it is not clear how do they define the mask $r$ in eq. (2) and (3). Is it user-defined, which requires additional supervision on every inference step? If so, it seems to be unfair to compare it with fully autonomous approaches like InterFaceGAN.
- line 258: FID is calculated with 400 edited images which seem to be very strange. One of FID calculation steps is the computation of a covariance matrix of Inception-V3 embeddings of the data. The matrix seems to be computed over just 400 points which is not enough and incorrect as the embeddings have dimensionality 2048. The common choice for FID computation is a tenth of thousands of samples.
- As the authors report that LocalEditing always demonstrates poor quality on the real data, there should be an ablation of different samples inversion approaches, e.g. e4e [3] that guarantees that an image inverts to an in-domain latent distribution.
- In Table 1 the authors miss a number of baselines, including [4] and [5]. [4] proposes to distill a GAN latent manipulation into a pix2pix model and apply it to real data. [5] localizes regions of latent manipulations with weak human supervision.
- It is not clear why authors incorporate a non-official GANLocalEditing implementation while the official can be found in the original paper [6].
- in Table 1 it is not clear how latent directions were scaled for different methods. For instance, Figure 10 suggests that there is a scale of the InterFaceGAN "smile" direction that demonstrates Attribute Accuracy close to 0.9. and ID Score close to 0.7. Once properly scaled, this direction could act superiorly compared to EditGAN in Table 1.
- As for me, it was hard to follow some parts of the text. For instance, eq. (1) operates with all symbols being defined below that makes it hard to read. Though the authors report that they can't share the code before publication, a pseudo-code could significantly clarify the approach. 
- line 239-240: "To the best of our knowledge, no previous methods can perform as complex and high-precision edits as we do, while preserving image quality and subject identity." this statement seems to be a bit speculative as there are no experiments that confirm it. The prior works have also demonstrated impressive editing results and the ability to combine edits.


[1] Datasetgan: Efficient labeled data factory with minimal human effort, Zhang et al.

[2] Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization, Li et al.

[3] Designing an Encoder for StyleGAN Image Manipulation, Tov et al.

[4] StyleGAN2 Distillation for Feed-forward Image Manipulation, Viazovetskyi et al.

[5] Mask-Guided Discovery of Semantic Manifolds in Generative Models, Yang et al.

[6] https://github.com/IVRL/GANLocalEditing

-----------------
upd: after rebuttal I raise my score 4 -> 6
Though I still think that the method have some limitations and Table 1 should be either removed or all the directions should be rescaled to match some of the parameters, I don't think that this should be considered as an obstacle for acceptance. Yes.

 We would like to follow up regarding your review and our response. We hope that we were able to address your questions in our response and that our extensive additional experiments address your concerns. We will incorporate all clarifications and additional results in the final version of the paper. If you feel that our response is satisfactory, we would like to kindly ask you to consider raising your paper rating accordingly. Otherwise, please let us know where you think we could further improve. Thank you very much!

This submission tackles the problem of semantic edition of images.
The proposed method builds heavily on DatasetGAN. 
To edit an image x, the method first finds the corresponding w in the GAN latent space (using a trained encoder) and then optimizes it so that the semantic branch output aligns to a target label map. The process effectively produces a modified image that retains the characteristics of the original image but with a semantic layout given by the target map.
The difference between the initial w and the optimized w provides an edit vector, that can be applied to other similar images to perform similar edits in real-time. If needed, some further optimization can still be performed to remove edition artifacts.  Originality:
The proposed way to perform semantic modification shares some similarity with [1]. I'm not sure such a recent paper does count as previous work, but then again, it is concurrent to DatasetGAN, on which the current submission is built upon.
In any case, the submission is still original as it also tackles the edition of real images using an encoder, as well as introducing edit vectors in this context.

Quality:
The submission is technically sound and well supported by empirical results.
Perhaps computing FID with only 400 images is somewhat debatable as FID can be unreliable with few data [2]. Using more images, or including other similar metrics [2] could strengthen the results in that regard.

Clarity: The submission is clear and well written.

Significance: Semantic edition is a challenging task that so far required specific pipelines.
This work demonstrates competitive performance for semantic edition while needing minimal labeling effort, and minimal additional training if a pre-trained GAN exists.
A most interesting find is that semantic edit vectors seem to transfer well between examples.
The main limitation of the method is that it is limited to the space of images that a GAN can generate.

Questions:
Can the authors confirm that the edit vector is computed using a single real image? (This might need to be explicitly stated in the paper.)
If that is the case, how is this image chosen, and how does this choice affect the quality of the edit vector? Also, how does the amplitude of the change in the target semantic map affect the edit vector?

---------------------------------------------------------------------------------------------------------------
Post-rebuttal:
I'd like to thank the authors for their responses and for providing additional results. My concerns have been adressed and I raised my score accordingly.
The proposed method is simple, with direct useful practical application. Moreover, I find the robustness of edit vectors and their applicability to different images particularly intriguing. I believe the implications regarding the good properties of StyleGAN-2's latent space can push our understanding of GANs forward and would be of interest to the community. Limitations and impacts are adequately addressed.

 __Regarding KID calculation:__ Thank you for pointing this out. We calculated the Kernel Inception Distance (KID) metric on the MaskGAN benchmark as suggested (using the same 400 test images as in the main paper here). The results are shown in the table below. The “ranking” between the different methods remains the same and consequently the conclusions do not change. In particular, EditGAN achieves the best attribute accuracies and ID scores, while InterfaceGAN has a slightly better KID and FID than EditGAN. We will report these numbers in the final version of the paper.

| Method | Attribute Accuracy (%) | FID | KID | ID score |
| --- | --- | --- | --- |  --- |
| MaskGAN | 77.3 | 46.84 | 0.020 | 0.4611 |
| LocalEditing | 26.0 | 41.26 | 0.012 | 0.5823 |
| InterfaceGAN | 83.5 | 39.42 | 0.010 |0.7295|
| EditGAN | 91.5 | 41.74 | 0.013 | 0.7047 |
| EditGAN + 30 refinement steps | 85.8 | 40.83 | 0.012 | 0.7452 |

__Regarding [1]:__ Thank you for bringing this work to our attention. The work shares important similarities in particular with DatasetGAN [2], on which our method builds. Furthermore, the paper explores a related editing approach as we do in our EditGAN as one of its applications. 
However, our editing approach is methodologically different and more advanced in multiple aspects:
1. [1] performs editing purely based on optimization. This can be problematic for practical real-time editing. We propose to amortize the optimization result into learnt editing vectors, which directly translate to new images and can be applied interactively in real-time. We will release an interactive web tool with a graphical user interface, where users can try out EditGAN themselves and edit images interactively.
2. When applying learnt editing vectors, we also propose to scale the edit with $s_\textrm{edit}$ to modulate its effect at test time. This provides additional editing flexibility and freedom at no additional cost. Leveraging our editing vector framework we can also easily combine different edits corresponding to different semantic changes at test time (Figs. 1 and 5). No such compositionality of edits is demonstrated by [1].
3. Furthermore, we propose a strategy for optional self-supervised refinement at test-time to carefully control editing quality when applying the editing vectors at test time. This allows us, if necessary, to remove editing artifacts that can occur for complex or large-scale editing operations.
4. We combine our framework with an additional encoder that allows us to edit real images that we embed into the GAN’s latent space.  

Furthermore, we demonstrate significantly more diverse and stronger experimental results:
1. [1] does neither demonstrate the extremely detailed, high precision edits that we are doing (modifying pupil sizes, changing gaze, rotating spoke, etc., Figs. 4 and 7), nor very large-scale edits (such as changing car type, Fig. 8).
2. [1] only operates on GAN-generated images. It is well-known that it is easier to edit GAN-generated images as opposed to embedded real ones. In contrast, we demonstrate all our edits in the paper on real embedded images.
3. [1] does not perform quantitative comparisons for their editing experiments. In our work, we thoroughly compare our method to multiple important and widely used baselines, demonstrating our superior editing performance.

Finally, as was pointed out by the reviewer, [1] only appeared on arXiv relatively shortly before the NeurIPS submission deadline. Consequently, we consider [1] as concurrent work with us.  
However, we will of course incorporate this paper in the discussion of related works in the final version of the paper.

[1] Jianjin Xu, Changxi Zheng, Linear Semantics in Generative Adversarial Networks. CVPR, 2021.  
[2] Zhang et al., DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort. CVPR, 2021.

 After reading the reponse, I realized my official review was missing the needed references.
I sincerely apologize for this mistake and would like to thank the authors for putting their best efforts into their response nonetheless.

The references that were missing from the original post are the following:

[1] Linear Semantics in Generative Adversarial Networks.
Jianjin Xu, Changxi Zheng. CVPR 2021

[2] Demystifying MMD GANs.
Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, Arthur Gretton. ICLR 2018

If possible, it would be useful if you could comment on similarities with [1] and provide KID scores proposed in [2].

 [1] MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. CVPR, 2020.  
[2] Sean: Image synthesis with semantic region adaptive normalization. CVPR, 2020.  
[3] Designing an Encoder for StyleGAN Image Manipulation. arXiv:2102.02766, 2021.  
[4] StyleGAN2 Distillation for Feed-forward Image Manipulation.arXiv:2003.03581, 2020.  
[5] Mask-Guided Discovery of Semantic Manifolds in Generative Models. arXiv:2105.07273, 2021.  

 We thank the reviewer for the thoughtful feedback and for acknowledging EditGAN’s elegance, simplicity and its strong qualitative results.  
__1. EditGAN’s applicability only to edits based on segmentations:__ EditGAN focuses on high-precision semantic editing driven by segmentation masks, where it outperforms all existing methods regarding the achievable level of detail (also see point 10 below). Editing operations involving, for example, color or texture variations or re-lighting are beyond our scope. We believe that there is not one single algorithm that is best suited for all kinds of edits. Our method is aligned with existing works that tackle these other types of edits. Our goal with EditGAN is to fill the current gap in the literature in very high-precision, real-time semantic editing while relying on little training data.  
__2. Editing different image parts corresponding to same segmentation class:__ Thank you for this suggestion. We tried the edit, i.e. independently modifying the size of the front and back wheels. We can confirm that we are indeed able to do this. We will include visualizations in the final manuscript.  
__3. Definition of mask $r$ for area of interest:__ The editing region $r$ is chosen fully automatically. Given a new image, the segmentations are automatically generated via the segmentation branch once the image is embedded in latent space. Furthermore, as discussed below Eq. (1), each edit has an edit-specific pre-specified list of part labels $Q_\textrm{edit}$ relevant for the edit (for example, we know class IDs for car wheels). Hence, given the predicted segmentation of the image to be edited, the editing region $r$ simply consists of those areas in the image where the segmentation label prediction matches a label from the list $Q_\textrm{edit}$; In conclusion, the selection of $r$ when editing new images is fully automatic. Hence, all comparisons to InterfaceGAN are fair.  
__4. FID calculation:__ We agree that an FID calculation with only 400 images can lead to biased estimates. We performed this evaluation on the existing benchmark proposed by MaskGAN [1], which enables a fair comparison with existing approaches (Table 1). MaskGAN itself used only 400 images for FID calculation, which we followed. Although the estimates may be biased, they should nevertheless enable fair comparisons among the different methods.  
To address the reviewer’s concern to our best effort, we re-calculated FID and the other metrics using 10 times as many images, i.e. 4000, from the training set from MaskGAN. Note that only MaskGAN uses this data for training, while the GANs of all other baselines, including EditGAN, are based on the FFHQ faces data and do not use this annotated training data that MaskGAN relies on. Hence, calculating the FID using these 4000 images is advantageous for MaskGAN.  
Results are in the table below. Since we now use different and much more data for evaluation compared to the previous evaluation, the numbers are different. However, the rankings and comparisons between the methods remain the same. In particular, EditGAN achieves the best attribute accuracies and ID scores, while InterfaceGAN has slightly better FID than EditGAN. MaskGAN achieves low FID, but this is due to the unfair comparison, as discussed above. MaskGAN still performs much worse than InterfaceGAN and EditGAN in Attribute Accuracy and ID score. Overall, the conclusions are the same.    

| Method                        | Attribute Accuracy (%) | FID   | ID score |
| ----------------------------- | ---------------------- | ----- | -------- |
| MaskGAN                       | 65.7                   | 18.27 | 0.5229   |
| InterfaceGAN                  | 79.5                   | 27.02 | 0.6560   |
| EditGAN                       | 88.0                   | 28.14 | 0.6422   |
| EditGAN + 30 refinement steps | 81.4                   | 27.66 | 0.6625   |

We did not report LocalEditing, since it requires optimization-based embedding of all 4000 test as well as reference images into latent space. We did not have the necessary computational resources for that available within the last week.    
__5. E4E encoder:__ We tried with [3] for LocalEditing and indeed found a significant improvement in attribute accuracy and ID score, but a slightly worse FID score. 

| Method                          | Attribute Accuracy (%) | FID   | ID score |
| ------------------------------- | ---------------------- | ----- | -------- |
| LocalEditing                    | 26.0                   | 41.26 | 0.58     |
| LocalEditing - Encoding4Editing | 41.75                  | 48.28 | 0.66     |
| EditGAN                         | 91.5                   | 41.74 | 0.70     |
| EditGAN + 30 refinement steps   | 85.8                   | 40.83 | 0.75     |

Note that there is a trade-off between perceptual quality, distortion, and editability when using the e4e encoder, which is nicely discussed here https://github.com/omertov/encoder4editing/issues/2. Importantly, EditGAN still generally outperforms LocalEditing, even when LocalEditing uses the e4e encoder.  
__6. Additional baselines:__ Thank you for pointing out these additional works. While [5] allows exploring plausible image completions by smoothly traversing StyleGAN’s latent space, it does not allow defining specific edits as necessary for the benchmark. Hence, we cannot use this method for quantitative comparisons. We will include it in our related work section.  
However, we did run [4] for the smile benchmark and obtained the following results (evaluation based on the 400 MaskGAN test set images):  

| Method                        | Attribute Accuracy (%) | FID   | ID score |
| ----------------------------- | ---------------------- | ----- | -------- |
| StyleGAN2 Distillation        | 98.3                   | 45.09 | 0.78     |
| EditGAN                       | 91.5                   | 41.74 | 0.70     |
| EditGAN + 30 refinement steps | 85.8                   | 40.83 | 0.75     |

We see that [4] performs well, although FID is a bit worse. However, the contribution of StyleGAN2 Distillation is it creates an alternative approach that doesn't need real image embedding and it relies on editing-vector model (like us and InterfaceGAN) to create a training dataset. Hence, in the paper, relying on pre-trained classifier to find editing vector, the paper only allows relatively high-level editing of image attributes for which large-scale annotations exist. They can not perform the high-precision editing that our EditGAN can do, such as gaze changes, pupil dilation, etc. Moreover, a new Pixel2PixelHD network needs to be trained for each edit, limiting broad, user-interactive applicability.
However, there might be an interesting synergy: We could use our EditGAN framework to find new, highly detailed edits with our segmentation-driven approach, and then use the StyleGAN2 Distillation framework to distill those into a pix2pixHD network. Hence, we consider the two approaches as complementary.  
__7. LocalEditing implementation:__ This is a typo that happened during the rush for paper submission. We actually *did* use the original LocalEditing repository, i.e. https://github.com/IVRL/GANLocalEditing. We are sorry about this and will correct this.   
__8. Editing scales for different methods:__ For the results in Table 1, for all methods that involve scalings of the edits, a unit scale was used. Generally, different scales will result in different trade-offs regarding attribute accuracy and identity preservation and consequently there is not “one best scale”. This is why we also have Figure 10, where we varied the editing scalings both for InterfaceGAN and EditGAN over a broad range. We find that EditGAN and InterfaceGAN perform similarly when we do not use any self-supervised refinements in EditGAN, while our refinement approach leads to better trade-offs. The main advantage of EditGAN over InterfaceGAN is that EditGAN requires much fewer labels for training and enables much higher precision editing. InterfaceGAN relies on high-level attribute classifiers, which are pre-trained using large annotated datasets, and cannot perform high-precision edits.  
__9. Readability and symbol definitions:__ Thank you for pointing out that Eq. (1) was difficult to parse. We will rearrange this paragraph and define the relevant symbols earlier. We will also include pseudo-code in the next paper version. Furthermore, if accepted we will release code and an interactive online tool for users to explore EditGAN.  
__10. Regarding Claims:__  We agree the prior works have demonstrated impressive editing results and the ability to combine edits. However to the best of our knowledge, EditGAN indeed achieves editing with a higher level of detail than any existing methods. This is in part because EditGAN requires only a handful of segmentation annotations during training. Consequently, we can leverage segmentation annotation schemas that are annotated with an extremely high level of detail (see Fig. 6 in the main paper and Fig. 1 in the Supp. Mat.), but for which no large-scale annotated data sets exist. For example, for faces the annotations we can use have details such as eye ball annotations and even pupils. This allows us to perform extremely high precision editing, such as changing the gaze, modifying specific wrinkles, or modifying pupil size (see Figures 4, 5, and 7). 

__Final Remarks:__ We hope that we were able to clarify some aspects of EditGAN and eliminate concerns. We will include all additional baselines in the final version of the paper. Please let us know if you have further questions or concerns. Thank you very much.  


 We would like to thank the reviewer for the constructive feedback and the positive evaluation of our manuscript.  
__Regarding running-times:__ We have carefully measured the runtime of our model. Conditional optimization, given an edited segmentation mask, with 30 (60) optimization steps takes around 11.4 (18.9) seconds. This operation provides us the editing vector $\delta \mathbf{w}^{+}_{edit}$. Application of editing vectors is almost instantaneous, taking only 0.4 seconds, therefore allowing for complex real-time interactive editing. A 10 (30) step self-supervised refinement would add an additional 4.2 (9.5) seconds.  
We will incorporate a table with these numbers in the next version of the manuscript. We would also like to point out that these running-times are based on our implementation of EditGAN for research purposes. An optimized implementation of EditGAN for production purposes can likely reduce these numbers significantly.  
__Regarding comparisons to other GAN architectures:__ This is an excellent question. Our EditGAN approach relies on features learnt by the GAN that are also useful for segmentation as well as on good embedding methods. Generally, we believe that such features should also be found in other types of architectures, such as BigGAN. Embedding methods and encoder architectures have been developed primarily for StyleGAN, but we do not see fundamental reasons why BigGAN should not allow for something similar. That being said, we very much agree that it is unclear whether BigGAN features something similar to StyleGAN’s $\mathcal{W}$-space (or $\mathcal{W}^{+}$), which is highly useful and leveraged in EditGAN.   
Generally, BigGAN is in particular known to perform well for modeling diverse data such as ImageNet, while StyleGAN seems to be particularly well suited for high-resolution and detailed modeling of more uni-modal images such as faces. Consequently, we believe that when applying our framework on data such as ImageNet, it might be beneficial to try building on top of BigGAN. We think that this is a very important and promising future research direction, in particular with regards to further scaling our EditGAN framework to even more complex and diverse data.  
__Regarding other conditioning methods:__ We agree that our approach will potentially also work with conditioning data other than semantic segmentations and we would like to thank the reviewer for the suggestions. We are indeed planning to investigate keypoint-based editing, and we similarly envision text-based and reference image-based editing and learning of editing vectors within future versions of our framework. These are important directions for future research. In the current work, our goal was to achieve very high-precision semantic editing, which seems challenging with many previous editing methods, as discussed in the paper. To this end, leveraging detailed semantic segmentations in the EditGAN framework seemed ideal---hence our focus on this modality.   

In summary, we strongly agree: Our EditGAN approach is general and we expect it to be extended to other conditioning data in the future. 
Please let us know if you have further questions about our method. Thank you very much.



 We would like to thank the reviewer for their thoughtful feedback and the positive evaluation of our manuscript.  
__Regarding FID calculation:__ We sympathize with the reviewer’s comment that an FID calculation with only 400 images can potentially lead to biased FID estimates. We performed this evaluation on the existing benchmark proposed by MaskGAN [1], which enables a fair comparison with existing approaches (Table 1). Note that MaskGAN itself used only 400 images for FID calculation, which we followed. Although the estimates may be biased with respect to the true FID, they should nevertheless enable a fair comparison among the different methods.  
In an effort to address the reviewer’s concern to our best effort, we re-calculated FID as well as the other metrics, Attribute Accuracy and ID score, using 10 times as many images, i.e. 4000 images, from the training set from MaskGAN. Notice that only MaskGAN uses this data for training, while the GANs of all other baselines, including our EditGAN, are based on the FFHQ faces data and do not use this annotated training data that MaskGAN relies on. Hence, calculating the FID using these 4000 images is advantageous for MaskGAN.
The results are shown in the table below. Since we use different and much more data for evaluation compared to the evaluation reported in the table in the paper, the numbers are different. However, the rankings and comparisons between the methods remain the same and the conclusions are the same. In particular, EditGAN achieves the best attribute accuracies and ID scores, while InterfaceGAN has a slightly better FID than EditGAN. MaskGAN achieves a very low FID, but this is simply due to the unfair comparison, as discussed above. MaskGAN still performs significantly worse than InterfaceGAN and EditGAN in Attribute Accuracy and ID score. Overall, the conclusions are the same. We will report these numbers in the final version of the paper.

| Method | Attribute Accuracy (%) | FID | ID score |
| --- | --- | --- | --- |
| MaskGAN | 65.7 | 18.27 | 0.5229 |
| InterfaceGAN | 79.5 | 27.02 | 0.6560 |
| EditGAN | 88.0 | 28.14 | 0.6422 |
| EditGAN + 30 refinement steps | 81.4 | 27.66 | 0.6625 |

Note that we did not report LocalEditing here, since running LocalEditing requires optimization-based embedding of all 4000 images as well as the reference images into the GAN latent space, which is very time consuming to run and we did not have the computational resources available within the last week.  
__Regarding Editing Vector Learning from Single Images:__ Yes, we can confirm that editing vectors are created from single images. We will better clarify this in the final version of the paper. The specific images that were used for all edits demonstrated in the paper as well as the specific segmentation modifications to learn the editing vectors are shown in the Supplemental Material in Figures 2 and 5-21. These images (and corresponding segmentations) were randomly chosen from the small annotated training set for the semantic segmentation branch. The choice itself did not have a large effect on the final edit so long as the editing area was clearly visible (when editing wheel size of a car, for example, it is beneficial if the wheel is clearly visible, which would not be the case if we see the car directly from the front).   
The amplitude of the change in the target segmentation map can have an effect on the editing vector, too. On the one hand, a very small change in the segmentation would not result in a strong signal during optimization when learning the editing vector, which can result in an imprecise editing vector. On the other hand, a very large change in the segmentation may push the edit too far and correspond to a highly unrealistic output, far from anything the GAN has seen during training. This can lead to artifacts in the edit which are then encoded in the editing vector. Consequently, when learning editing vectors we use segmentation changes that are large, but still correspond to plausible and realistic segmentations (see Figures 2 and 5-21 in the Supplemental Material).  
Please let us know if you have further questions about our EditGAN. Thank you very much.  
[1] Cheng-Han Lee et al., MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. CVPR, 2020.

The paper proposes to perform image editing via segmentation maps. First, a joint representation of image and segmentation map is learned with a GAN such that the GAN's latent representation w represents both the image and the segmentation map. This can be done with very few labeled datapoints. After this, a given image can be edited by optimizing the latent representation w to represent the edited segmentation map, which will implicitly also make the generated image follow the edited segmentation map. The authors perform all their experiments with the StyleGAN architecture and show that their approach works for several domains such as faces, cars, cats, etc. and in high resolution (up to 1024 px). Compared to previous methods this approach can do very fine-grained edits, the edits are much better disentangled, and only very few labeled samples are needed for training.  The paper is well written and shows compelling results. The approach offers several advantages over previous editing approaches (high disentanglement, supervised edits, few labeled data needs, etc). The evaluation is well done and extensive, comparing to several baselines and running it on several different domains.

It would be helpful if there was some more information about running-times of the individual steps that need to be done, i.e. how long (roughly) does it take to find the optimized delta-w for a given edit and how long does the self-supervised refinement take (when it is necessary)?

One question I have is whether or how well this approach will work with other GAN architectures, e.g. BigGAN. Do you have any initial results or intuition here? Since e.g. BigGAN does not have the w-space for "easy" optimization given an edited segmentation map I wonder how well this method would transfer?

Also, have you thought of ways to incorporate other conditioning methods for editing into this framework, e.g. text-based commands, keypoints, reference images, etc? The authors address the societal impact in the sense that this approach is one of several that can enable easy edits of images, possibly amplifying the problems of fake news or unreliable data online.

Meta-review: This submission tackles the problem of semantic editing of images leveraging a generative model that modulates a joint distribution of images with segmentation masks. Thanks to this joint generation, the editing approach is based on the segmentation masks manipulation.  
This allows highly localized editing that were difficult to obtain up to now. 

Some issues and/or limitations have been raised by reviewers. Many clarifications have been done by authors during rebuttal and discussion period. Scores have been increased. 

The task, the novel methodology, and the quality of the experiments with direct useful practical application lead me to propose to accept this paper.

Decision: accept