EditGAN: High-Precision Semantic Image Editing
Authors: Huan Ling
Karsten Kreis
Daiqing Li
Seung Wook Kim
Antonio Torralba
Sanja Fidler
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative adversarial networks
image editing

Abstract: Generative adversarial networks (GANs) have recently found applications in image editing. However, most GAN-based image editing methods often require large-scale datasets with semantic segmentation annotations for training, only provide high-level control, or merely interpolate between different images. Here, we propose EditGAN, a novel method for high-quality, high-precision semantic image editing, allowing users to edit images by modifying their highly detailed part segmentation masks, e.g., drawing a new mask for the headlight of a car. EditGAN builds on a GAN framework that jointly models images and their semantic segmentation, requiring only a handful of labeled examples – making it a scalable tool for editing. Specifically, we embed an image into the GAN’s latent space and perform conditional latent code optimization according to the segmentation edit, which effectively also modifies the image. To amortize optimization, we find “editing vectors” in latent space that realize the edits. The framework allows us to learn an arbitrary number of editing vectors, which can then be directly applied on other images at interactive rates. We experimentally show that EditGAN can manipulate images with an unprecedented level of detail and freedom while preserving full image quality. We can also easily combine multiple edits and perform plausible edits beyond EditGAN’s training data. We demonstrate EditGAN on a wide variety of image types and quantitatively outperform several previous editing methods on standard editing benchmark tasks.

EditGAN: High-Precision Semantic Image Editing
Huan Lingl
122,,
Karsten Kreis!
Daiqing Li
Seung Wook Kim 1,2.3
Antonio Torralba
Sanja Fidler
'VVIDIA University of Toronto
Vecttor Institute
MIT
(hul ing, kiree a, dal ging), seungwookk, sflderrt w a.com torra laagmit..eds
Abstract
Generative adversarial networks (GANs) have recently found applications in im- age editing. However, most GAN-based image editing methods often require large-scale datasets with semantic segmentation annotations for training. only provide high level control, merely interpolate between different images. Here, we propose EditGAN, novel method for high-qualite image editing, allowing users to edit images by modifying their highly detailed part segmentation masks, e.g., drawing new mask for the headlight of a car. EditGAN builds on a GAN framework that jointly models images and their seman- tic segmentations [1, 2]. requiring only handful of labeled examples making it scalable tool for editing. Specifically, we embed an image into the GAN's latent space and perform conditional latent code optimization according to the segmentation edit, which effectively also modifies the image. timization, find "editing vectors" in latent space that realize the edits. The framework allows us to learn an arbitrary number of editing vectors, which can then be directly applied other images at interactive rates. We experimentally show that EditGAN can manipulate images with an unprecedented level of detail and freedom, while preserving full image quality. We can also easily combine multiple edits and perform plausible edits beyond EditGAN's training data. We demon- strate EditGAN on wide variety of image types and quantitatively outperform several previous editing methods on standard editing benchmark tasks. Project page: https: /ttp///vvttttt tlabs. gi github.ttt. io/edi tGAN.
semantic
amortize op-
1 Introduction
Change Shape
Enlarge Wheels
Shrink Front-Light
Al-driven photo and image edit- ing has the potential to streamline the workflow of photographers and content creators and to enable new levels of creativity and dig- ital artistry [3]. Al-based image editing tools have already found their way into consumer software in the form of neural photo edit- ing filters, and the deep learning research community is actively developing further techniques. A particularly promising line of research uses generative adversarial networks (GANs) [4, 5, 6, 7, 8] and either embeds images into
Figure 1: High-precision semantic image editing with EditGAN.
'These authors contributed equally.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

-1- Train EdiGGAN
4- Real-time Fditing with Editing Vectors
W
G:W+ + X
-2 Edit Segmetttation Mask
Enlarge
-3- Learn Editing Vector in Latent Space
Wheols!
8w
Figure (I) EdiGAN builds on GAN framework that jointly models images and their semantic segmentations. (2 & 3) Users can modify segmentation masks, based on which we perform optimization in the GAN's latent space to realize the edit. (4) Users can perform editing simply by applying previously Icartt editing vectors and manipulate images at interactive rates.
the GAN's latent space or works directly with GAN- -enerated images. Careful modifications of the latent embeddings then translate to desired changes in generated output, allowing, for example, to coherently change facial expressions in portraits [9, 10, 11, 12, 13. 14, 15, 16], change viewpoint or shapes and textures of cars [17], or to interpolate between different images in a semantically meaningful manner [18, 19, 20, 21].
Most GAN-based image editing methods fall into few categories. Some works rely on GANs
conditioning on class labels or pixel-wise semantic segmentation annotations [19, 10, 22, 11], where different conditionings lead to modifications in the output, while others use auxiliary attribute classifiers [23, 15] to guide synthesis and edit images. However, training such conditional GANs or external classifiers requires large labeled datasets. Therefore, these methods are currently limited to image types for which large annotated datasets are available, like portraits [10]. Furthermore, even if annotations are available, most techniques offer only limited editing control, since these annotations usually consist only of high-level global attributes or relatively coarse pixel-wise segmentations. Another line of work focuses on mixing and interpolating features from different images [18, 19, 20, 21], thereby requiring reference images as editing targets and usually also not offering fine control. Other approaches carefully analyze and dissect GANs' latent spaces, finding disentangled latent variables suitable for editing [24, 25, 12, 13, 14, 26, 27], or control the GANs' network parameters [25, 28, 16]. Usually, these methods do not enable detailed editing and are often slow.
In this work, we are addressing these limitations and propose EdirGAN, novel GAN-based image editing framework that enables high-precision semantic image editing by allowing users to modify detailed object part segmentations. EditGAN builds on a recently proposed GAN that jointly models both images and their semantic segmentations based on the same underlying latent code [1, 2], and requires as few as 16 labeled examples allowing it to scale to many object classes and choices of part labels. We achieve editing by modifying the segmentation mask according to desired edit and optimizing the latent code to be consistent with the new segmentation, thus effectively changing the RGB image. To achieve efficiency, we learn editing vectors latent space that realize the edits, and that can be directly applied on other images, without any only few additional optimization steps. We can thus pre-train a library of interesting edits that a user can directly utilize an interactive tool.
We apply EditAAN on a wide range of images, including images of cars, cats, birds, and human faces. demonstrating unprecedented high-precision editing. We perform quantitative comparisons to multiple baselines and outperform them in metrics such as identity preservation, quality preservation, and target attribute accuracy, while requiring orders of magnitude less annotated training data, EditGAN is the first GAN-driven image editing framework, which simultaneously (i) offers very high- precision editing, (ii) requires only very little annotated training data (and does not rely on external classifiers), (iii) can be run interactively in real time, (iv) allows for straightforward compositionality of multiple edits, (y) and works on real embedded, G-N-generated, and even out-of-domain images.
2 Related Work
Image Editing and Manipulation. Image Editing has a long history in computer vision and graphics,
as well as machine learning [29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 18, 11, 28, 41, 42, 16). Recently, deep generative models [4, 43, 44], in particular modern GANs [6, 45, 7, 46, 8], received

---

much attention as a promising tool for efficient image editing, as it was found that latent space manipulations often lead interpretable and predictable changes in output [47, 24, 48, 49, 26, 27, 50].
GAN-based image editing methods can be broadly sorted into a number of categories. (i) One line of work relies on the careful dissection of the GAN's latent space, aiming find interpretable and disentangled latent variables, which can be leveraged for image editing, in a fully unsupervised manner [47, 24, 25, 12, 13, 14, 48, 49, 26, 27, 50, 51]. Although powerful, these approaches usually do not result in any high-precision editing capabilities. The editing vectors we are learning in EditGAN would be too hard to find independently without segmentation based guidance. (ii) Other works utilize GANs that condition on class or pixel-wise semantic segmentation labels to control synthesis and achieve editing [9, 52, 46, 19, 10. 22, 11]. Hence, these works usually rely on large annotated datasets, which are often not available, and even if available, the possible editing operations are tied to whatever labels are available. This stands in stark contrast to EdiGGAN, which can be trained in semi-supervised fashion with very little labeled data and where an arbitrary number of high-pree eigh-precsioo edits can be learnt. (iii) Furthermore, auxiliary attribute classifiers have been used for image manipulation [23, 15]. thereby still relying annotated data and usually only providing high-level control. (iv) Image editing is often explored in the context of "interpolating" between a target and different reference image in sophisticated ways, for example by replacing certain features in a given image with features from a reference images [18, 19, 20, 21]. From the general image editing perspective, the requirement of reference images limits the broad applicability of these techniques and prevents the user from performing specific, detailed edits for which potentially no reference images are available. (v) Recently, different works proposed to directly operate in the parameter space of the GAN instead of the latent space to realize different edits [25, 28, 16]. For example, [25, 28] essentially specialize the generator network for certain images test time aid image embedding or "rewrite" the network to achieve desired semantic changes in output. The drawback is that such specializations prevent the model from being used in real-time on different images and with different edits. [16] proposed an approach that more directly analyses the parameter space of a GAN and treats it as latent space in which apply edits. However, the method still merely discovers edits in the network's 's parameter space, rather than actively defining them like we do. It remains unclear whether their method can combine multiple such edits, as we can, considering that they change the GAN parameters themselves. (vi) Finally, another line of research targets primarily very high-level image and photo stylization and global appearance modifications [37. 53, 54, 55, 52, 56, 46, 57, 41].
Generally, most works only do relatively high-level and not the detailed, nigh-precision editing, which EditGAN targets. Hence, we consider EditGAN as complementary to this body of work.
GANS and Latent Space Image Embedding. EditGAN builds on top of DataserGAN [1] and SemanticGAN [2], which proposed to jointly model images and their semantic segmentations using shared latent codes. However, these works leveraged this model design only for semi-supervised learning, not for editing. EditGAN also relies on an encoder, together with optimization, to embed new images to be edited into the GAN's latent space. This task in itself has been studied extensively in different contexts before, and we are building these works. Previous papers studied encoder- based techniques [63, 64, 65, 66, 67, 68,
methods [58, 59, 60, 61, 62), used primarily optimization-basee 69, 26], and developed hybrid approaches [63, 24, 25, 70, 71].
Finally, concurrent paper [72] shares similarities with DatasetGAN [1], on which our method builds, and explores editing approach related to our EditGAN as one of its applications. However, our editing approach is methodologically different and leverages editing vectors, and also demonstrates significantly more diverse and stronger experimental results. Furthermore, [73] shares some high- level ideas with EditGAN; however, it leverages the CLIP [74] model and targets text-driven editing.
3 High-Precision Semantic Image Editing with EditGAN
3.1 Background
EditGAN's image generation component is StyleGAN2 [7, 8), currently the of-the-r-t GAN for image synthesis. The StyleGAN2 generator maps latent codes z E Z, drawn from multivariate Normal distribution, into realistic images. A latent code z is first transformed into an intermediate code w E W by a non-linear mapping function and then further transformed into K + 1 vectors, w , w through learned affine transformations. These transformed latent codes are fed into synthesis blocks, whose outputs are deep feature maps.
3

---

Deep generative models such as StyleGAN2, which are trained to synthesize highly realistic images, acquire a semantic understanding of the modeled images in their high- nigh-dimensional feature space. Recently, DatasetGAN [1] and SemanticGAN [2] built on this insight to learn a joint distribution p(x,y) y) over images x and pixel-wise semantic segmentation labels y, while requiring only a handful of labeled examples. EditGAN utilizes this joint distribution y(x,y) to perform high-precision semantic image editing of real and synthesized images.
Both methods [1, 2] model p(x,y) y) by adding an additional segmentation branch to the image generator, which is pre-trained StyleGAN [1]. We follow DatasetGAN [1], which applies a simple three-layer multi-layer perceptron classifier on the layer-wise concatenated and appropriately
upsampled feature maps. This classifier operates on the concatenated feature maps in a per-pixel fashion and predicts the segmentation label of each pixel.
3.2 Segmentation Training and Inference by Embedding Images into GAN's Latent Space
To both train the segmentation branch and perform segmentation on a new image, we embed an image into the GAN's latent space using an encoder and optimization. To this end, we build on previous works [66, 62, 2] and train an encoder that embeds images into W space, which is defined as W but where the w's are modeled independently [66, 62]. Our objectives to train this encoder consist of standard pixel-wise L2 and perceptual LPIPS reconstruction losses using both the real training data as well samples from the GAN itself. For the GAN samples, we also explicitly regularize the encoder with the known underlying latent codes. In practice, we use the encoder to initialize images' latent space embeddings and then iteratively refine the latent code w via optimization, again using standard reconstruction objectives.
In that way, we embed the annotated images x from dataset labeled with semantic segmentations into latent space, and train the segmentation branch of the generator using standard supervised learning objectives, i... the cross entropy loss. We keep the image generator's weights frozen and only backpropagate the loss to the segmentation branch [1]. After training the segmentation branch, we can formally define a generator G : W + X that models the joint distribution p(x. y) of images and semantic segmentations y. Details about encoder and segmentation branch training as well as optimization for image embedding can be found in the Appendix.
3.3 Finding Semantics in Latent Space via Segmentation Editing
The key idea of EditGAN lies in leveraging the joint distribution p(x,y) y) of images and semantic seg- mentations for high-precision age editing. Given a new image x to be edited, we can embed it into EditGAN's W* latent space, described above (alternatively, we can also sample images from the model itself and use those). The seg- mentation branch will then generate the corresponding segmentation y, since segmentations and RGB im-
Vow LeB
Oytimization in Latest Space
W
im-
V Vww CRGB
We modify semantic segmentations and optimize the shared latent code for consistency with the new segmentation within the editing region, and with the RGB appearance outside the editing region. Corresponding gradients are backpropagated through the shared generator. The result is latent space editing vector ow
Figure 3:
ages share the same latent codes w Using simple interactive digital painting or labeling tools, we can now manually modify the segmentation according tation mask by Yedied. Starting from the embedding t of the unedited image x and segmentation y, we can then perform optimization within W to find a new w Wedited w + dw odit consistent with the new segmentation Yeditd. while allowing the RGB output x to change within the editing region.
desired edit. We denote the edited segmen-
Formally, we are seeking an editing vector 5 E W such that (Xedned. Yedited = Ō(( +5w ) edit where G denotes the fixed generator that synthesizes both images and segmentations. Defining (x','' = G(w + dw+), we perform optimization to approximate dw edit by 8w+. The region of interest within which we expect the image to change due to the edit is formally given by
{p : c Qodit) U {p= c Vine
Qedit)
which means that is defined by all pixels p whose part segmentation labels op yynut according to either the initial segmentation y the edited one Yedited are within an edit-specific pre-specified list

---

Qudit of part labels relevant for the edit. For example, when modifying the wheel in a photo of a car Qedit would contain all part labels related to the wheels, such as tire, spoke, and wheelhub (see Fig. 3). We use a further buffer of 5 pixels to give the GAN freedom in modeling the transition between the edited and non-edited area. In practice, r acts as a binary pixel-wise mask (see Eqs. and below).
Note that Xedited is not available during optimization. After all, Xedited is the edited image we are ultimately intested in. It emerges indirectly when optimizing for the segmentation modification, since images and segmentations are closely tied together in the joint distribution p(x. modeled by G. We further define x' (w + ow+) as G's image generation and y' (P'W"" + +wwt) G's segmentation generation branch.
To find ow approximating 8w ean we use the following losses minimization targets:
CROB(óW*) ) = LLPIS (G" + tow") @ (1 (1-11) r), x x (1 -r))
(2)
+ ( + ) (1 r), x o (1 r))
CCE(Öw*) = H (w+ + +ww+) Or,, Yedited r)
(3)
where H denotes the pixel-wise cross-entropy, LIPIP loss is based on the Learned Perceptual Image Patch Similarity (LPIPS) distance [75], and LL2 is a regular pixel-wise L2 loss. LRGB (ōw+) ensures that the image appearance does not change outside the region of interest. while LCE(Öw ensures that the target segmentation Yedited is enforced within the editing region (see visualization in Fig. 3). When editing human faces, we also apply the identity loss [62]]
(iw +) = (R(C** (w" + dw )), R(x))
(4)
with R denoting the pretrained ArcFace feature extraction network [76] and (;,)) cosine-similiarity
The final objective function for optimization then becomes:
Lediting (5w+) = ccditing LRGB (öw+) + Aditing x (ow( + Age oditing (1w(5
(5)
with hyperparameters > The only "learnable" variable the editing vector ow+; all neural networks are kept fixed. After optimizing 5w+ with the objective function, we can use 8w+ A= 8w edit Note that there a certain amount of ambiguity in how the segmentation modification is realized in RGB output. We rely on the GAN generator, trained to synthesize realistic images, to modify the RGB values in the editing region in a plausible way consistent with the segmentation edit.
3.4 Different Ways of Editing during Inference
The latent space editing vectors ow sdit obtained by optimization as described are semantically meaningful and often disentangled with other attributes. Therefore, for new images x to be edited, we can embed the images into the W latent space and the same editing operations can be directly performed by applying the previously learnt tw as (x', y') G(w+ Sedu 8w edu without doing any optimization from scratch again. In other words, the learnt editing vectors dw+ amortize the
iterative optimization that was necessary to achieve the edit initially. For well-disentangled editing operations, x' can be used directly as the edited image Xefited Note that we introduced Sedit, scalar editing coefficient, which effectively scales and controls the editing magnitude during inference, For Sedit O, we do not do any editing all, while for Sodit 1 manipulate the images with an effectively larger editing operation in latent space, leading to exaggerated effects.
Unfortunately, disentanglement is not always perfect and the editing vectors dw do not always translate perfectly to other images. We can remove editing artifacts in other regions of the image by a few additional optimization steps at test time. Specifically, we can use the exact same minimization objectives as above, using the initial prediction y', obtained after applying the editing vector dw edit as Yedited This assumes that the editing vector still induces a plausible segmentation change when applied on other images and that artifacts only arise in RGB output. The RGB objective LRGB then removes these editing artifacts outside the editing region, while LeE ensures that the modified segmentation stays as predicted by the editing vector.
+
Summarizing, we can perform image editing with EditGAN in three different modes:
Real-time Editing with Editing Vectors. For localized, well-disentangled edits we perform editing purely by applying previously learnt editing vectors with varying scales Sedit and manipulate images at interactive rates.
5

---

De-wriakle
Close Eyes
Gaze Position
Gaze Pos.
Hairstyle
Raise Byetrows
Smile
Enlarge Wheels
Shrink Wheels
Enlarge Front Light
Add Lioense-Plate
Remove Side Mirror
Enlarge Eyes
Enlarge Ear
Smaller Ear
Longer Beak
Head Up
Bigger Belly
Figure 4: Examples of segmentation-driinn edits with EdiGAN. Results are based on editing with editing vectors and 30 steps self-supervised refinement. Blue boxes: Original images. Orange hoxes: 7oom-in views.
Vector-based Editing with Sele-Supe Refinement. For localized edits that are not perfectly disentangled with other parts of the image, we can remove editing artifacts by additional optimization at test time, while initializing the edit using the learnt editing vectors. Appimizattonnbasse Editing, Image-specific and very large edits do not transfer other images via editing vectors. For such operations, we perform optimization from scratch.
4 Experiments
We extensively evaluate EditGAN on images across four different categories: Cars (384x512 spatial resolution), Birds (512x512), Cats (256x256) and Faces (1024x 1024).
Implementation We train our segmentation branch as described in Sec. 3.2 using 16, 16, 30, and 30 image-mask pairs labeled training data for Faces, Cars, Birds, and Cats, respectively. We utilize very highly-detailed part segmentations from [1]. The annotation scheme for faces shown Fig. 7, all others are presented in the Appendix. When editing done purely optimization-basee or when learning the editing vectors, we always perform 100 steps of optimization using Adam [77]. For Car, Cat, and Faces, use real images from DatasetGAN's test that were not part of GAN training to demonstrate editing functionality. These images are first embedded into EditGAN's latent space via an encoder and optimization described in Sec. 3.2. For Birds, we show editing on -en-ggeeeateed images. Model details and hyperparameters are provided in the Appendix.
4.1 Qualitative Results
In-Domain Results In Fig. 4, we demonstrate our EditGAN framework when applying previously learnt editing vectors 8w edit novel images and refining with 30 steps of optimization. Our editing operations preserve high image quality and are well disentangled for all classes. We also show ability to combine multiple different edits in Fig. 5. To the best of our knowledge, methods can perform as complex and high-precision edits as we do, while preserving image quality and subject identity. In Fig. 8, we demonstrate that
previous
perform extremely high-precision

---

+ Open Eyes
+ Close Mouth
+ Look Right
er
er
Figure 5: We combine multiple edits. Results are based editing with editing vectors and 30 steps self- supervised refinement. Blue boxes: Original images. Edits in detail: Second row, first person: open eyes, add hair, mustache. Second person: smile, left. Third row, first car: remove mirror, remove door handle, shrink wheels. Second car: remove license plate, enlarge wheels. Third row, bird: longer beak, bigger belly, head up. Third row, cat: open mouth, bigger ear. bigger eyes.
edits, such as rotating a car's wheel spoke or dilating pupils. EditGAN can edit semantic parts of objects that consist of only few pixels. At the same time, we can use EditGAN to perform large-scale modifications, too: Fig. 9, we present how we can remove the entire roof car or convert it to station wagon-like vehicle, simply by modifying the segmentation mask accordingly and optimizing. is worth noting that several of our editing operations generate plausible manipulated images unlike those appearing in the GAN training data. For example, the training data does not include cats with overly large eyes or ears. Nevertheless, we achieve such edits a sigh-quality manner.
The edits in Figs. 4, 5 and 8 are based learnt editing vectors with self- self-supervised refinement. However, without such refinement usually only very minor artifacts occur, as shown Fig. 10, hence allowing for real-time high-precision semantic image editing (discussed in detail below).
Out-of-Domait Results We demonstrate the generalization capability of EditGAN out-of- domain data on the MetFaces [8] data set. We use our EditGAN model trained on FFHQ [8], and create editing vectors 8wi eda using in-domain real faces. We then embed out-of-domain MetFaces partraits (with 100 steps optimization) and apply the editing vectors with 30 steps self-supervised refinement. The results are shown in Fig. 6. We find that our editing operations seamlessly translate even to such far out-of-domain examples.
4.2 Quantitative Results
To quantitatively measure EditGAN's image editing capabilities, we use the smile edit benchmark introduced by MaskGAN [10]. Faces with neutral expressions are converted into smiling faces and performance is measured by three metrics: a. Semantic Correctness: Using a pre-trained smile attribute classifier, we measure whether the faces show smiling expressions after editing. b. Distribution-level Image Quality: Frechet Inception Distance (FID) [78, 79] and Kernel Inception Distance (KID) [80] calculated between 400 edited test images and the CelebA-HD test dataset.
7

---

images. Results are based on editing with editing vectors and 30 steps self- supervied refinement. Edits in detail: First row, first example: look left, frown. Second example: smile, look right. Second row, first example: open eyes, lift eyebrow. Second example: open eyes.
Figure 6: We combine multiple edits on
Using the pretrained ArcFace feature extraction network [76], we measure whether the subjects' identity is maintained when applying the edit. Specifically, we report cosine- similiarity between original and edited images. Further details can be found in the Appendix.
Identity Preservation:
For our EditGAN, we simply learn a smil- ing editing vector dw edi using a hold-out neutral expression face image. We embed it into EditGAN, infer its pixel-wise seg- mentation labels, and manually modify the segmentation towards smile. Then we perform optimization in latent space, as de- scribed above, to learn the editing vector. For the results in Tab. I, it is applied with unit scale =cdi=1 on new images. We do
Maric
A Mask a Altibate Attnibttc FID KID ID Scree
Annr Annet.
MasKGAN [10]
20,000
77.5
46.84
0,000
0.4611
Localidiing (18)
76.0
41.76
0013
0.5835
LocalEdiing- EnodinneBidnn [811
41.75
48.29
0,016
0.5603
IateFaasGAN [13]
30,000
83.5
39.42
0,810
0,7295
EdRGAN (ours)
16
91.5
41.74
0.013
0,7047
EdiMIAN" 30 (ours)
16
85.8
40.83
0.022
0.7452
StyleGANZ Distilation
30000
0,7825
Table 1:
Quantitative comparisons multiple baselines the smile edit benchmark.
not use the identity loss (Bq. 4) in this experiment, since identity preservation is already a target method with three strong baselines: (i) MaskGAN2 [10]: It takes
metric itself. We compare non-smiling images, their segmentation masks, and a target smiling segmentation mask as inputs. Note that training MaskGAN requires large annotated datasets, in contrast to us. We also compare to (ii) LocalEditing' [18]: It clusters GAN features to achieve local editing and relies on reference images, in this case images of faces with smiling expressions. Another baseline we use is (iii) InterFaceGANt [13]: Similar to EditGAN, InterFaceGAN aims at finding editing vectors in latent space. However, uses auxiliary attribute classifiers, relies on large annotated datasets, and can generally not achieve the fine editing control of our Edi(GAN. Finally, we compare to (iv) StyleGAN2 Distillation [82], which creates an alternative approach that does not require real image embeddings and also relies on an editing- editing-vector vector model to create a training dataset.
Results are reported in Tab. 1. Using 1, x less training labels, we outperform MaskGAN all three metrics. We similarly obtain significantly stronger results than LocalEditing. In our observation, LocalEdittng does work well real image embeddings. We further exploit better encoder [81] for the LocalFditing baseline, which leads to a significant improvement in attribute accuracy and ID score, but slightly worse FID & KID scores. We find that EditGAN outperforms INterFaceGAN on identity preservation and attribute classification accuracy, while InterFaceGAN reaches slightly better FID & KID scores (for the results in Tab. the latent space edits learnt by InterfaceGAN are also applied with unit scale, like for EditGAN). In Fig. 11, we report a more detailed comparison to InterFaceGAN, where we apply the smile editing vectors with different scale coefficients from zero to two. As shown, when the editing vector scale is small, the identity score is high while the smiling attribute score low, since the modification of the original images is minimal. We find that our real- time editing with editing vectors is on-par with InterFaceGAN, When we perform supf-vupd

---

Sybrr
Figure 8: High-pree figh-precision editing with EditGAN for extreme details. Left: We rotate spoke. Right: We modify pupil size. Results are based on editing with editing vectors and 30 steps self-supervised refinement.
Figure Face part beling schemna [1].
Figure 9: Pure optimization-bn editing. We demonstrate large-scale semantic edits that do not transfer scamlossly to other images via editing vectors. Hence, we perform ontimization from scratch.
Buitian int a-
UcEng Scling
Figure 10: Left: We apply learnt editing vectors with varying scales (see markers in FID plots) both without (top row for each class) and with (bottom row each class) additional 30-step supervipe refinement correct artifacts. Red boxes denote original images. For each class, the leftmost image is the one used to leam the editing vector, with the editing result next to it and orginal and modified segmentations below. Right: Visual quality after editing with different scales measured by FID with and without refinement.
refinement at test time, EditGAN outperforms InterFaceGAN. In Tab. 1, we also compare with StyleGAN2 Distillation [82]. which achieves strong performance. However, StyleGAN2 Distillation relies on pre-trained classifiers, like InterfaceGAN, and only enables relatively high-level editing of image attributes for which large-scale annotations exit. Moreover, it distills edits into separate Pixe|2PixelHD networks, such that new network needs to be trained for each edit, limiting broad, user-interactive applicability. Hence, we consider StyleGAN2 Distillation orthogonal to our EditGAN.
Running Time
We carefully measure the run time of our editing on an NVIDIA Tesla V100 GPU. Conditional optimization, given an edited segmentation mask, with 30 (60) optimization steps takes 11.4 (18.9) seconds. This operation provides us the editing vector. Application of editing vectors is almost instantaneous, taking only 0.4 seconds, therefore allowing for complex real-time interactive editing. A 10 (30) step self-supervised refinement would add an additional 4.2 (9.5) seconds.
4.3 Ablation Studies: Self-Supervised Refinement and Editing Vector Scale
Fig. 11 also contains quantitative ablation study on the number of additional optimization steps done when initializing edit with learnt editing vector and refining with additional optimization. Generally, the more refinement steps we perform, the better the performance our model can achieve. As shown in Fig. 11, find that further optimization can indeed slightly improve performance. Specifically, here we improve the trade-off between maintaining identity and achieving the desired se- mantic operation when performing editing with different scalings Seda of the editing vector. However, performing many steps of optimization leads to a run-time vs. performance trade-off, and our results suggest that the improvement beyond 30 additional optimization steps becomes marginal.
In Fig. 10, we analyze the editing vector scale and self-supervised refinement visually and with respect to perceptual metrics. As highlighted in the zoom-in areas, small artifacts appear due

---

to imperfect disentanglement in latent space when applying editing operations with large scales. self-supervised refinement successfully cleans these editing errors up. We also apply the same edit with different scales on 400 test images and measure FID with respect to 10,000 data from GAN training, inspired by the analyses in [16]. We can clearly see that image quality degrades as measured by FID, the stronger the edit is applied. We also observe small improvements with the iterative refinement on this metric, although the difference is small. Further details are in the Appendix. We conclude that for most editing operations, real-time editing without iterative refinement already performs very well. However, to clean up artifacts and maintain highest image quality possible, self- supervise refinement with couple of additional optimization steps is always available.
Additional experiments are presented in the Appendix.
0.315
0.750
5 Conclusions
Limitations Like all GAN-based image editing meth- ods, EditGAN is limited to images that can be modeled by the GAN. This makes EditGAN's application on, for in- stance, photos of vivid city scenes challenging. Although most of our high-precision edits readily transfer to other images via learnt editing vectors, also encountered challenging edits that required iterative optimization on each example. Future research therefore includes speeding up the optimization for such edits as well as building better generative models with more disentangled latent spaces.
00873
InafeceSsy
0.650
0.625
Ednsin Aafreman:: 36
CSNSAN tefremenc 10
6.55
6.30 100
Figure 11: InterFaceGAN's and EditGAN's performance on the smile edit benchmark for different editing vector scalings (scale in- creases from top-left points towards bottom- right points; see main text and Appendix for details). For EditGAN, we optionally add 10, 30 or 60 additional optimization steps.
Summary We propose EditGAN, novel method for high-precision high-quality semantic image editing. It relies on a GAN that jointly models RGB images and their pixel-wise semantic segmenta- tions and that requires only very few annotated data for training. Editing is achieved by performing optimization in latent space while conditioning on edited segmentation masks. This optimization can be amortized into editing vectors in latent space, which can be applied on other images directly, allowing for real-time interactive editing without any or only little further optimization. We demon- strate a broad variety of editing operations on different kinds of images, achieving an unprecedented level of flexibility and freedom in terms of editing, while preserving high image quality.
Broader Impact
Where previous generative modeling based image editing methods offer only limited high-level editing capabilities, our method provides users unprecedented
semantic editing possi- bilities. Our proposed techniques can be used for artistic purposes and creative expression and benefit designers, photographers, and content creators [3]. Al-driven image editing tools like ours promise to democratize high-quality image editing. Related methods have already found their way into everyday applications in the form of neural photo editing filters. On a larger scale, the ability to synthesize data with specific attributes can be leveraged in training and finetuning machine learning models.
At the same time, more precise photo editing also offers opportunities for advanced photo manipula- tion for nefarious purposes. The recent progress of generative models and Al-driven photo editing has profound implications on image authenticity and beyond, which is an area of active debate [83]. As one potential way to tackle these challenges. methods for automatically validating real images and detecting manipulated or fake images are being developed by the research community [84, 85]. Furthermore, generative models like ours are usually only as good as the data they were trained on. Therefore, biases in the underlying datasets are still present in the synthesized images and preserved even when applying our proposed editing methods. It is therefore important to be aware of such biases in the underlying data and counteract them, for example by actively collecting more representative data or by using bias correction methods, an area of active research [86, 87, 88, 89].
Funding Statement
This work was funded by NVIDIA. Huan Ling and Seung Wook Kim acknowledge additional revenue in the form of student scholarships from University of Toronto and the Vector Institute, which are not in direct support of this work.
10

---

References
[1] Yuxuan Zhang. Huan Ling, Jun Gao, Kangxue Yin, Sean-Francoif Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labcled data factory with minimal human effort. arXiv preprint Xiv: 06490640,,0.....11111 2021.
[2] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba. and Sanja Fidler. Semantic segmentation with generative models: Sumi-supe learning and strong out-of-domain generalization. arXiv preprint arXiv:2104.05833 2021.
[3] J. Bailey. The tools of generative art, from flash neural networks. Art in America, 2020.
[4] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[5] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 1rXiv:/5/1.0645553 2015.
[6] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint htt.:::1110.119969. 2017.
[7] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[8] Tero Karras, Samuli Laine. Mïka Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 81 11008199, 2020.
[9] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Won Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-imaae translation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018
[10] Cheng Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In JEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[11] Rongliang Wu, Gongjie Zhang, Shijian Lu, and Tao Chen. Cascade ef-gan: Progressive facial expression editing with local focuses. In Proceedings of the IELE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[12] Yujun Shen. Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. CVPR, 2020.
[13] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. TPAMI, 2020.
[14] Yazeed Alharbi and Peter Wonka. Disentangled image generation through structured noise injection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[15] Xianxu Hou, Xiaokang Zhang, Linlin Shen, Zhihui Lai, and Jun Wan. Guidedstyle: Attribute knowledge guided style manipulation for semantic face editing. arXiv preprint arXiv:2012.11856, 2020.
[16] Anton Cherepkov. Andrey Voynov, and Artem Babenko. Navigating the gan parameter space for semantic image editing. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). 2021.
[17] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering, arXiv preprint arXiv:20/0.091 2020.
[18] Edo Collins, Raja Bala, Bob Price, and Sabine Süsstrunk. Editing in style: Uncovering the local semantics of GANs. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[19] Peihao Zhu, Ramcen Ahdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region- adaptive normalization. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). June 2020.
[20] Kathleen M Lewis. Srivatsan Varadharajan, and Ira Kemelmacher-Shhizzennnn Vogue: Try-on by stylegan interpolation optimization. arXiv preprint arXiv:2101.00 022800 2021.
11

---

[21] Hyunsu Kim. Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Uh. Exploiting spatial dimensions of latent in gan for real-time image editing. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2021.
[22] Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongho Fu, Deepfacedrawing: Deep generation of face images from sketches. ACM Trans. Graph. 39))., 2020.
[23] Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen. Atlgan: Facial attribute editing by only changing what you want. IEEE Transactions on Image Processing, 28(1 (8(11)554445477, Nov 2019.
[24] David Bau, Jun Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[25] David Bau, Hendrik Strobelt. William Peebles, Jonas Wulff. Bolei Zhou, Jun- Yan Zhu, and Antonio Torralba. Semantic photo manipulation with generative image prior. ACM Trans. Graph., 38(4). 2019.
[26] Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. In International Conference on Learning Representations, 2020.
[27] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. In Proc. NeurlPS, 2020.
[28] David Bau, Steven Liu,, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba. Rewriting a deep generative model. In Proceedings of the European Conference on Computer Vision (ECCV). 2020.
[29] George Walherg. Digital Image Warping. TEEE Computer Society Press, Washington, DC, USA, 1st. edition, 1994.
[30] Alexei A. Efros and William T. Freeman. Image quilting for texture synthesis and transfer. SIGGRAPH '01, page 341-346, New York, NY, USA, 2001. Association for Computing Machinery.
[31] Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salcsin. Image analo- gies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '01, page 327-340, New York, NY, USA, 2001. Association for Computing Machinery.
[32] F.. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley. Color transfer between images. TEEE Computer Graphics and Applications, 21(5):34 41, 2001.
[33] Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing. SIGGRAPH '03, page 313-318, New York, NY, USA, 2003, Association for Computing Machinery.
[34] Scott Schacfer, Travis McPhail, and Joc Warren. Image deformation using moving least squares. ACM Trans. Gruph., 25(3):333-540) 2006.
[35] Connelly Barnes, Eli Shechtman. Adam Finkelstein. and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3), 2009.
[36] Michael W. Tao, Micah K. Johnson, and Sylvain Paris. Error-tolerant image compositing, In ECCV, 2010.
[37] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR). 2016.
[38] Jun-Yan Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaited image-to laage-to-ime to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[39] Tiziano Portenier, Qiyang Hu, Attila Szabó, Siavash Arjomand Bigdeli, Paolo Favaro, and Matthias Zwicker. Faceshop: Deep sketch-based face image editing. ACM Trans. Graph., 37(4), 2018
[40] Huan Ling, David Acuna, Karsten Kreis, Seung Wook Kim, and Sanja Fidler. Variational amodal object completion. Advances in Neural Information Processing Systems, 2020.
[41] Taesung Park, Jun- Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexci Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. In Advances in Neural Information Processing Systems, 2020.
[42] Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. DriveGAN: Towards a Controllable High- Quality Neural Simulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2021.
12

---

[43] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In The International Conference Learning Reprexentations (ICLR), 2014.
[44] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approx- imate inference in deep generative models. In International Conference Machine Learning, pages 1278-1286, 2014.
[45] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations. 2019.
[46] Taesung Park, Ming-Yi Yu Liu. Ting-Chun Wang, and Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- rion, pages 2337-2346, 2019.
[47] Lore Goetschalekx, Alex Andonian. Aude Oliva, and Phillip Isola. Ganalyze: Toward visual definitions of cognitive image propertics. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), Octoher 2019.
[48] Ali Jahanian*, Lucy Chai*, and Phillip Isola. On the "steerability" of generative adversarial networks. In International Conference on Learning Representations, 2020.
[49] Andrey Voynov and Artem Babenko. Unsupervised discovery interpretable directions in the gan latent space. In International Conference on Machine Learning, pages 9786-9796. PMLR, 2020.
[50] Binxu Wang and Carlos R Ponce, A geometric analysis deep generative image models and its applications. In International Conference on Learning Representations, 2021.
[51] Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In CVPR, 2021.
[52] Ting-Chun Wang, Ming- Yu Liu, lun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 8798-8807, 2018.
[53] Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[54] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-lo-ing to-image translation networks, In Advances in neural information processing systems, pages 700-708, 2017.
[55] Yijun Li, Ming- Yu Liu. Xueting Li, Ming Hsuan Yang, and Jan Kautz. A closed-form solution to photorealistic image stylization. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[56] H. Kazemi, S. Iranmanesh, and N. Nasrabadi. Style and content disentanglement in generative adversarial networks. In 2019 JEEE Winter Conference on Applications of Computer Vision (WACV), pages 848 856, Los Alamitos, CA. USA, jan 2019. IEEE Computer Society.
[57] Jacjun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung- Woo Ha. Photorcalistic style transfer via wavelet transforms. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), 2019.
[58] Guim Perarnau, Joost van de Weijer, Bogdan Raducanu, and Jose M. Álvarez. Invertible conditional gans for image editing. arXiv preprint 1.06355, 2016.
[59] Jeff Donahue, Philipp Krähenbühh,, and Trevor Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.099 1005.09885. 2016
[60] Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[61] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martín Arjovsky, Olivier Mastropietro, and Aaron C. Courville. Adversarially learned inference. In 5th International Conference on Learning Repre- sentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[62] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: stylegan encoder for image-to to-inage translation. arXiv preprint 2000:2008.0095) 2020.
13

---

[63] Jun- Yan Zhu, Philiip Krähenbühl. Eli Shechtman, and Alexei A Efros, Generative visual manipulation on the natural image manifold. In European conference on computer vision, pages 597-613. Springer, 2016.
[64] R. A. Yeh, C. Chen, T. Y. Lim, A. G. Schwing. M. Hasegawa-Johnsse and M. N. Do. Semantic image inpainting with deep generative models. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6882-6890, 2017.
[65] Zachary C. Lipton and Subarna Tripathi. Precise recovery latent vectors from generative adversarial networks. arXiv preprini 1rXivv:702.04777 2017.
[66] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE International Conference on Computer Vision, pages 4432 4441, 2019.
[67] Minyoung Huh, Richard Zhang, Yan Zhu, Sylvain Paris, and Aaron Hertzmann. Transforming and projecting images into class-conditional generative networks. arXiv preprint arXiv:2005.07003, 2020.
[68] A. Creswell and A. A. Bharath. Inverting the generator of generative adversarial network. IEEE Transactions on Neural Networks and Learning Systems, 30(7):1977-1974 2019.
[69] A. Raj. Li, and Y. Brester. Gan-based projector for faster recovery with convergence guarantees in linear inverse problems, In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5601-5610, 2019.
[70] D. Bau, J. Zhu, J. Wulff, W. Peebles, B. Zhou, H. Strobelt, and A. Torralba. Seeing what a gan cannot generate. In 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pages 4501-455 2019.
[71] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. arXiv preprint arXiv:2004.00 00049, 2020.
[72] Jianjin Xu and Changxi Zheng, Linear semantics in generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9351-9360, 2021.
[73] David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba. Paint by word. arXiv preprint arXiv:2103.10951 2021.
[74] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint ar arXiv:2103 0000000000 2021.
[75] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[76] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafciriou. Areface: Additive angular margin loss for deep face recogmition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2019.
[77] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint MIXiv:1412.6980. 2014.
[78] Maximilian Seitzer, pytorch-fid: FID Score for PyTorch. pytorch- fid, August 2020. Version 0.1.1.
[79] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a time-scale update rule converge to local nash equilibrium. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and Garnett, Advances in Neural Information Processing Systems 30, pages 6626-6637. Curran Associates, Inc., 2017.
[80] Mikotaj Binkowski, Danica J. Sutherland, Michael Arhel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018
[81] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1-14, 2021.
[82] Yuri Viazovetskyi. Vladimir Ivashkin, and Evgeny Kashin. Stylegan2 distillation for feed-forwar forward image manipulation. In European Conference Computer Vision, pages 170-186. Springer, 2020.
14

---

[83] Cristian Vaccari and Andrew Chadwick. Deepfakes and disinformation: Exploring the impact of synthetic political video deception, uncertainty, and trust Social Media + Society, 6()):2056305 20003308, 2020.
[84] Thanh Thi Nguyen, Quoe Viet Hung Nguyen, Cuong M. Nguyen, Dung Nguyen, Duc Thanh Nguyen. and Saeid Nahavandi. Deep learning for deepfakes creation and detection: survey. arXiv preprint arXiv:1909.15525 2021.
[85] Yisroel Mirsky and Wenke Lee. The creation and detection of deepfakes: survey, ACM Comput. Surv.. 54( 2021.
[86] Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Eric Horvitz, and Stefano Ermon. Bias correction of learned generative models using likelihood importance weighting. In Advances in Neural Information Processing Systems, 2019.
[87] Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In Proceedings of the 37th International Conference on Machine Learning. 2020.
[88] Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz. Inclusive GAN: improving data and minority coverage in generative models. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII, 2020.
[89] Jinhee Lee. Haeri Kim, Youngkyu Hong, and Hye Won Chung. Self-diagnosing gan: Diagnosing underrepresented samples in generative adversarial networks, arXiv preprint arXiv:2102.1205. 2021.
15


Reviews:
[{"review_id": "Yt6_O20As2", "reviewer": "Reviewer_9mLQ", "summary": "The paper proposes a new method for semantic image editing, which allows users to edit images by changing their segmentation masks. More specifically, the method embeds an image required to be modified into the GAN's latent space and then perform optimisation according to the segmentation edit.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The proposed method builds on DatasetGAN and SemanticGAN, but allows users to achieve image editing. The difference between proposed method and previous works have been shown in related work.\n\nQuality: The submission is technically sound, claims are well supported, and authors have discussed strengths and weaknesses of the proposed method.\n\nClarity: The paper is well organised and easily followed.\n\nSignificance: The results look good, which achieve high-quality image editing. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations: \n1. The proposed editing method relies heavily on the shared latent code between images and their segmentation masks. However, image and segmentation masks are different modal information, so how do authors ensure these two features are exactly match in latent space?\n\n2. The method depends on the quality of segmentation masks, so if the generated masks fail to separate each part of an object, can proposed method still modify desired regions? e.g., in cat and bird examples, the proposed method seems to have limited modification choices.\n\n3. Can proposed method control the scale of modified effect?\n\n4. The proposed method builds on top of DatasetGAN, relying on modifying generated segmentation masks. Although authors show some differences in line 102, it might be better to discuss more about their differences or highlight their contribution.\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Vtrqx1zm6S0", "reviewer": "Reviewer_zqNn", "summary": "The authors proposed a local image-editing algorithm based on pre-trained StyleGAN2. The motivation is simple. Editing is performed based on two branches: generation and parsing. Parsing relies on pre-defined segmentation masks. The incremental editing vector is learnt by constraints to edited images and associated parsing masks. To handle the disentanglement, additional optimization is needed to perform refinement. ", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is based on previous works: DatasetGAN and SemanticGAN. It can be viewed as a specific application of these two works on part editing.  But the improvement of the algorithm for image editing is nearly trivial. Besides, the similar idea is also applied in the following paper\n\nPaint by word\nhttps://arxiv.org/pdf/2103.10951.pdf\n\nReplacing the NLP part with semantic masks in the above paper, the remaining core idea of solving editing vectors is nearly the same. Compared with Image2StyleGAN and Image2StyleGAN++, the improvement is the parsing branch, which is the idea of DatasetGAN and SemanticGAN. So, the novelty of the idea is too trivial compared with existing methods. I don't think the current state of the paper is suitable for NeurIPS publication. \n\n\n%%%%%% comments after rebuttal %%%%%%\n\nThank you for your detailed feedback. I agree with the authors that they proposed a simple method for local image editing. But the limited novelty is obvious. I still think it is obviously below the bar of NeurIPS acceptance. We need to encourage innovation. You may revise your paper according to Reviewers' concerns, especially Reviewer M7u3 who gave constructive advice from several different aspects.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "91N8KWOU0TY", "reviewer": "Reviewer_7mk5", "summary": "This paper proposed a semantic image editing method that allows users to edit an input image by manually edit the corresponding parts of its predicted semantic segmentation map.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this paper is incremental and does not have enough contributions to be published at top venues:\n1. This paper uses a combination of well-known methods: i) the joint modeling of images and their semantic segmentation [2][3] and ii) GAN inversion [4].\n2. This is not the first paper that enables high-precision semantic image editing, e.g. SEAN [1]. In addition, methods like SEAN are more flexible as they enable users to control the styles in each semantic region while the proposed method cannot.\n3. Compared to SEAN or similar methods, the major merit of the proposed method is that it uses less labeled data. However, this is a feature of previous studies [2][3].\n\nSo I think it is okay but not good enough.\n\n[1] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region adaptive normalization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.\n\n[2] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. arXiv preprint arXiv:2104.06490, 2021. \n\n[3] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. arXiv preprint arXiv:2104.05833, 2021.\n\n[4] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE International Conference on Computer Vision, pages 4432–4441, 2019.\n\n**Originality**: This paper tackles an existing task using a new combination of existing methods (i.e. joint modeling of images and their semantic segmentation [2][3], GAN inversion [4]). As abovementioned, the contributions are incremental.\n\n**Quality**: I think the methodology and claims of this paper are convincing as most of them are similar to those in [2][3] and [4]. The major problem is that it lacks new insights and is incremental.\n\n**Clarity**: The submission is clearly written and easy to follow. \n\n**Significance**: I think the results are okay but not good enough. As a combination of joint modeling of images and their semantic segmentation [2][3] and GAN inversion [4], the results are not surprising at all.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors discussed both limitations and societal impact.\nHowever, there is one missing limitation: the proposed method can only control the shape of the semantic editing regions but not their contents.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "008wbZRDFeN", "reviewer": "Reviewer_M7u3", "summary": "This work proposes EditGAN for semantic image editing with pre-trained GANs. It first introduces an additional branch beyond the original GAN generator to predict the semantic map corresponding to the image synthesis (this part is proposed by previous work). It then learns an encoder to project a target image to a latent code (this part is also proposed by previous work). At this end, given an image with the associated latent code and segmentation map, EditGAN enables semantic image editing by (1) modulating the segmentation map, (2) optimizing the latent code to fulfill the segmentation change, and (3) defining the semantic vector as the difference between the initial latent code and the optimized code. It turns out that such semantic vector is applicable to other images.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n\n- Good editing results, especially for the high-precision editing on details, such as spoke-rotation in Fig. 7.\n\n\nCons\n\n- Limited novelty. The generator (StyleGAN2), segmentation branch (DatasetGAN), GAN inversion (pSp) are all proposed by previous work. Editing the segmentation map is able to edit the image correspondingly is not that surprising. From this perspective, the technical novelty is incremental.\n\n- Lack of an important baseline. If I understand correctly, in the process of learning the edit vectors, users are required to modulate the segmentation map to make the modulation reasonable. For example, in Fig. 1, enlarging the wheels yet keeping the segmentation map reasonable is not that trivial. Then, a straightforward pipeline would be \"what if directly changing the image, like using PhotoShop?\". From my point of view, editing the image has the same difficulty as editing the segmentation map. In this way, we may not need the segmentation branch. More concretely, given an image A and an edited (with PhotoShop) image B, we can invert both of them to the latent space and get two latent codes, a and b. Then, (b-a) defines a semantic vector. According to my experience, such an operation is able to achieve wearing eyeglasses on face models. Without the comparison to such a baseline, I am not convinced that the proposed approach can give better results, but it requires much more effort than the aforementioned baseline.\n\n- Limited editing flexibility. \n(1) Is it possible to change the appearance of a particular segmentation part, e.g., changing the hair color? \n(2) Is it possible to make the person one eye looking left and the other right?\n(3) Enlarging ear and deleting ear in Fig. 4 do not make sense to me. Also, the ears are not fully deleted. \n(4) When shrinking a certain part, how to fill the missing pixels seems to be a problem. Taking the first car on the 3rd row of Fig. 5 as an example, the missing parts are blurry after shrinking the wheels.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper mentioned the broader impact in Sec.6.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method for GAN-based image editing with the pipeline where a user modifies a segmentation mask of an image. While the paper is well-written and the editing results are impressive, four reviewers unanimously consider the technical novelty of the paper as limited and the paper being incremental. Since none of the reviewers advocated for this work, my recommendation is "Reject".
Decision: Accept (Poster)