EditGAN: High-Precision Semantic Image Editing

1 Introduction Change Shape Enlarge Wheels Shrink Front-Light Frown Look Left Smile Figure 1 : High-precision semantic image editing with EditGAN . AI-driven photo and image editing has the potential to streamline the workflow of photographers and content creators and to enable new levels of creativity and digital artistry [ 3 ] . AI-based image editing tools have already found their way into consumer software in the form of neural photo editing filters , and the deep learning research community is actively developing further techniques . A particularly promising line of research uses generative adversarial networks ( GANs ) [ 4 , 5 , 6 , 7 , 8 ] and either embeds images into ∗These authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the GAN ’ s latent space or works directly with GAN-generated images . Careful modifications of the latent embeddings then translate to desired changes in generated output , allowing , for example , to coherently change facial expressions in portraits [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] , change viewpoint or shapes and textures of cars [ 17 ] , or to interpolate between different images in a semantically meaningful manner [ 18 , 19 , 20 , 21 ] . Most GAN-based image editing methods fall into few categories . Some works rely on GANs conditioning on class labels or pixel-wise semantic segmentation annotations [ 19 , 10 , 22 , 11 ] , where different conditionings lead to modifications in the output , while others use auxiliary attribute classifiers [ 23 , 15 ] to guide synthesis and edit images . However , training such conditional GANs or external classifiers requires large labeled datasets . Therefore , these methods are currently limited to image types for which large annotated datasets are available , like portraits [ 10 ] . Furthermore , even if annotations are available , most techniques offer only limited editing control , since these annotations usually consist only of high-level global attributes or relatively coarse pixel-wise segmentations . Another line of work focuses on mixing and interpolating features from different images [ 18 , 19 , 20 , 21 ] , thereby requiring reference images as editing targets and usually also not offering fine control . Other approaches carefully analyze and dissect GANs ’ latent spaces , finding disentangled latent variables suitable for editing [ 24 , 25 , 12 , 13 , 14 , 26 , 27 ] , or control the GANs ’ network parameters [ 25 , 28 , 16 ] . Usually , these methods do not enable detailed editing and are often slow . In this work , we are addressing these limitations and propose EditGAN , a novel GAN-based image editing framework that enables high-precision semantic image editing by allowing users to modify detailed object part segmentations . EditGAN builds on a recently proposed GAN that jointly models both images and their semantic segmentations based on the same underlying latent code [ 1 , 2 ] , and requires as few as 16 labeled examples – allowing it to scale to many object classes and choices of part labels . We achieve editing by modifying the segmentation mask according to a desired edit and optimizing the latent code to be consistent with the new segmentation , thus effectively changing the RGB image . To achieve efficiency , we learn editing vectors in latent space that realize the edits , and that can be directly applied on other images , without any or only few additional optimization steps . We can thus pre-train a library of interesting edits that a user can directly utilize in an interactive tool . We apply EditGAN on a wide range of images , including images of cars , cats , birds , and human faces , demonstrating unprecedented high-precision editing . We perform quantitative comparisons to multiple baselines and outperform them in metrics such as identity preservation , quality preservation , and target attribute accuracy , while requiring orders of magnitude less annotated training data . EditGAN is the first GAN-driven image editing framework , which simultaneously ( i ) offers very highprecision editing , ( ii ) requires only very little annotated training data ( and does not rely on external classifiers ) , ( iii ) can be run interactively in real time , ( iv ) allows for straightforward compositionality of multiple edits , ( v ) and works on real embedded , GAN-generated , and even out-of-domain images . 2 Related Work . Image Editing and Manipulation . Image Editing has a long history in computer vision and graphics , as well as machine learning [ 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 18 , 11 , 28 , 41 , 42 , 16 ] . Recently , deep generative models [ 4 , 43 , 44 ] , in particular modern GANs [ 6 , 45 , 7 , 46 , 8 ] , received much attention as a promising tool for efficient image editing , as it was found that latent space manipulations often lead to interpretable and predictable changes in output [ 47 , 24 , 48 , 49 , 26 , 27 , 50 ] . GAN-based image editing methods can be broadly sorted into a number of categories . ( i ) One line of work relies on the careful dissection of the GAN ’ s latent space , aiming to find interpretable and disentangled latent variables , which can be leveraged for image editing , in a fully unsupervised manner [ 47 , 24 , 25 , 12 , 13 , 14 , 48 , 49 , 26 , 27 , 50 , 51 ] . Although powerful , these approaches usually do not result in any high-precision editing capabilities . The editing vectors we are learning in EditGAN would be too hard to find independently without segmentation-based guidance . ( ii ) Other works utilize GANs that condition on class or pixel-wise semantic segmentation labels to control synthesis and achieve editing [ 9 , 52 , 46 , 19 , 10 , 22 , 11 ] . Hence , these works usually rely on large annotated datasets , which are often not available , and even if available , the possible editing operations are tied to whatever labels are available . This stands in stark contrast to EditGAN , which can be trained in a semi-supervised fashion with very little labeled data and where an arbitrary number of high-precision edits can be learnt . ( iii ) Furthermore , auxiliary attribute classifiers have been used for image manipulation [ 23 , 15 ] , thereby still relying on annotated data and usually only providing high-level control . ( iv ) Image editing is often explored in the context of “ interpolating ” between a target and different reference image in sophisticated ways , for example by replacing certain features in a given image with features from a reference images [ 18 , 19 , 20 , 21 ] . From the general image editing perspective , the requirement of reference images limits the broad applicability of these techniques and prevents the user from performing specific , detailed edits for which potentially no reference images are available . ( v ) Recently , different works proposed to directly operate in the parameter space of the GAN instead of the latent space to realize different edits [ 25 , 28 , 16 ] . For example , [ 25 , 28 ] essentially specialize the generator network for certain images at test time to aid image embedding or “ rewrite ” the network to achieve desired semantic changes in output . The drawback is that such specializations prevent the model from being used in real-time on different images and with different edits . [ 16 ] proposed an approach that more directly analyses the parameter space of a GAN and treats it as a latent space in which to apply edits . However , the method still merely discovers edits in the network ’ s parameter space , rather than actively defining them like we do . It remains unclear whether their method can combine multiple such edits , as we can , considering that they change the GAN parameters themselves . ( vi ) Finally , another line of research targets primarily very high-level image and photo stylization and global appearance modifications [ 37 , 53 , 54 , 55 , 52 , 56 , 46 , 57 , 41 ] . Generally , most works only do relatively high-level and not the detailed , high-precision editing , which EditGAN targets . Hence , we consider EditGAN as complementary to this body of work . GANs and Latent Space Image Embedding . EditGAN builds on top of DatasetGAN [ 1 ] and SemanticGAN [ 2 ] , which proposed to jointly model images and their semantic segmentations using shared latent codes . However , these works leveraged this model design only for semi-supervised learning , not for editing . EditGAN also relies on an encoder , together with optimization , to embed new images to be edited into the GAN ’ s latent space . This task in itself has been studied extensively in different contexts before , and we are building on these works . Previous papers studied encoder-based methods [ 58 , 59 , 60 , 61 , 62 ] , used primarily optimization-based techniques [ 63 , 64 , 65 , 66 , 67 , 68 , 69 , 26 ] , and developed hybrid approaches [ 63 , 24 , 25 , 70 , 71 ] . Finally , a concurrent paper [ 72 ] shares similarities with DatasetGAN [ 1 ] , on which our method builds , and explores an editing approach related to our EditGAN as one of its applications . However , our editing approach is methodologically different and leverages editing vectors , and also demonstrates significantly more diverse and stronger experimental results . Furthermore , [ 73 ] shares some highlevel ideas with EditGAN ; however , it leverages the CLIP [ 74 ] model and targets text-driven editing . 3 High-Precision Semantic Image Editing with EditGAN . 3.1 Background . EditGAN ’ s image generation component is StyleGAN2 [ 7 , 8 ] , currently the state-of-the-art GAN for image synthesis . The StyleGAN2 generator maps latent codes z ∈ Z , drawn from a multivariate Normal distribution , into realistic images . A latent code z is first transformed into an intermediate code w ∈ W by a non-linear mapping function and then further transformed into K + 1 vectors , w0 , ... , wK , through learned affine transformations . These transformed latent codes are fed into synthesis blocks , whose outputs are deep feature maps . Deep generative models such as StyleGAN2 , which are trained to synthesize highly realistic images , acquire a semantic understanding of the modeled images in their high-dimensional feature space . Recently , DatasetGAN [ 1 ] and SemanticGAN [ 2 ] built on this insight to learn a joint distribution p ( x , y ) over images x and pixel-wise semantic segmentation labels y , while requiring only a handful of labeled examples . EditGAN utilizes this joint distribution p ( x , y ) to perform high-precision semantic image editing of real and synthesized images . Both methods [ 1 , 2 ] model p ( x , y ) by adding an additional segmentation branch to the image generator , which is a pre-trained StyleGAN [ 1 ] . We follow DatasetGAN [ 1 ] , which applies a simple three-layer multi-layer perceptron classifier on the layer-wise concatenated and appropriately upsampled feature maps . This classifier operates on the concatenated feature maps in a per-pixel fashion and predicts the segmentation label of each pixel .

Summary: The paper proposes to perform image editing via segmentation maps. First, a joint representation of image and segmentation map is learned with a GAN such that the GAN's latent representation w represents both the image and the segmentation map. This can be done with very few labeled datapoints. After this, a given image can be edited by optimizing the latent representation w to represent the edited segmentation map, which will implicitly also make the generated image follow the edited segmentation map. The authors perform all their experiments with the StyleGAN architecture and show that their approach works for several domains such as faces, cars, cats, etc. and in high resolution (up to 1024 px). Compared to previous methods this approach can do very fine-grained edits, the edits are much better disentangled, and only very few labeled samples are needed for training.