EditGAN: High-Precision Semantic Image Editing

1 Introduction Change Shape Enlarge Wheels Shrink Front-Light Frown Look Left Smile Figure 1 : High-precision semantic image editing with EditGAN . AI-driven photo and image editing has the potential to streamline the workflow of photographers and content creators and to enable new levels of creativity and digital artistry [ 3 ] . AI-based image editing tools have already found their way into consumer software in the form of neural photo editing filters , and the deep learning research community is actively developing further techniques . A particularly promising line of research uses generative adversarial networks ( GANs ) [ 4 , 5 , 6 , 7 , 8 ] and either embeds images into ∗These authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the GAN ’ s latent space or works directly with GAN-generated images . Careful modifications of the latent embeddings then translate to desired changes in generated output , allowing , for example , to coherently change facial expressions in portraits [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] , change viewpoint or shapes and textures of cars [ 17 ] , or to interpolate between different images in a semantically meaningful manner [ 18 , 19 , 20 , 21 ] . Most GAN-based image editing methods fall into few categories . Some works rely on GANs conditioning on class labels or pixel-wise semantic segmentation annotations [ 19 , 10 , 22 , 11 ] , where different conditionings lead to modifications in the output , while others use auxiliary attribute classifiers [ 23 , 15 ] to guide synthesis and edit images . However , training such conditional GANs or external classifiers requires large labeled datasets . Therefore , these methods are currently limited to image types for which large annotated datasets are available , like portraits [ 10 ] . Furthermore , even if annotations are available , most techniques offer only limited editing control , since these annotations usually consist only of high-level global attributes or relatively coarse pixel-wise segmentations . Another line of work focuses on mixing and interpolating features from different images [ 18 , 19 , 20 , 21 ] , thereby requiring reference images as editing targets and usually also not offering fine control . Other approaches carefully analyze and dissect GANs ’ latent spaces , finding disentangled latent variables suitable for editing [ 24 , 25 , 12 , 13 , 14 , 26 , 27 ] , or control the GANs ’ network parameters [ 25 , 28 , 16 ] . Usually , these methods do not enable detailed editing and are often slow . In this work , we are addressing these limitations and propose EditGAN , a novel GAN-based image editing framework that enables high-precision semantic image editing by allowing users to modify detailed object part segmentations . EditGAN builds on a recently proposed GAN that jointly models both images and their semantic segmentations based on the same underlying latent code [ 1 , 2 ] , and requires as few as 16 labeled examples – allowing it to scale to many object classes and choices of part labels . We achieve editing by modifying the segmentation mask according to a desired edit and optimizing the latent code to be consistent with the new segmentation , thus effectively changing the RGB image . To achieve efficiency , we learn editing vectors in latent space that realize the edits , and that can be directly applied on other images , without any or only few additional optimization steps . We can thus pre-train a library of interesting edits that a user can directly utilize in an interactive tool . We apply EditGAN on a wide range of images , including images of cars , cats , birds , and human faces , demonstrating unprecedented high-precision editing . We perform quantitative comparisons to multiple baselines and outperform them in metrics such as identity preservation , quality preservation , and target attribute accuracy , while requiring orders of magnitude less annotated training data . EditGAN is the first GAN-driven image editing framework , which simultaneously ( i ) offers very highprecision editing , ( ii ) requires only very little annotated training data ( and does not rely on external classifiers ) , ( iii ) can be run interactively in real time , ( iv ) allows for straightforward compositionality of multiple edits , ( v ) and works on real embedded , GAN-generated , and even out-of-domain images . 2 Related Work . Image Editing and Manipulation . Image Editing has a long history in computer vision and graphics , as well as machine learning [ 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 18 , 11 , 28 , 41 , 42 , 16 ] . Recently , deep generative models [ 4 , 43 , 44 ] , in particular modern GANs [ 6 , 45 , 7 , 46 , 8 ] , received much attention as a promising tool for efficient image editing , as it was found that latent space manipulations often lead to interpretable and predictable changes in output [ 47 , 24 , 48 , 49 , 26 , 27 , 50 ] . GAN-based image editing methods can be broadly sorted into a number of categories . ( i ) One line of work relies on the careful dissection of the GAN ’ s latent space , aiming to find interpretable and disentangled latent variables , which can be leveraged for image editing , in a fully unsupervised manner [ 47 , 24 , 25 , 12 , 13 , 14 , 48 , 49 , 26 , 27 , 50 , 51 ] . Although powerful , these approaches usually do not result in any high-precision editing capabilities . The editing vectors we are learning in EditGAN would be too hard to find independently without segmentation-based guidance . ( ii ) Other works utilize GANs that condition on class or pixel-wise semantic segmentation labels to control synthesis and achieve editing [ 9 , 52 , 46 , 19 , 10 , 22 , 11 ] . Hence , these works usually rely on large annotated datasets , which are often not available , and even if available , the possible editing operations are tied to whatever labels are available . This stands in stark contrast to EditGAN , which can be trained in a semi-supervised fashion with very little labeled data and where an arbitrary number of high-precision edits can be learnt . ( iii ) Furthermore , auxiliary attribute classifiers have been used for image manipulation [ 23 , 15 ] , thereby still relying on annotated data and usually only providing high-level control . ( iv ) Image editing is often explored in the context of “ interpolating ” between a target and different reference image in sophisticated ways , for example by replacing certain features in a given image with features from a reference images [ 18 , 19 , 20 , 21 ] . From the general image editing perspective , the requirement of reference images limits the broad applicability of these techniques and prevents the user from performing specific , detailed edits for which potentially no reference images are available . ( v ) Recently , different works proposed to directly operate in the parameter space of the GAN instead of the latent space to realize different edits [ 25 , 28 , 16 ] . For example , [ 25 , 28 ] essentially specialize the generator network for certain images at test time to aid image embedding or “ rewrite ” the network to achieve desired semantic changes in output . The drawback is that such specializations prevent the model from being used in real-time on different images and with different edits . [ 16 ] proposed an approach that more directly analyses the parameter space of a GAN and treats it as a latent space in which to apply edits . However , the method still merely discovers edits in the network ’ s parameter space , rather than actively defining them like we do . It remains unclear whether their method can combine multiple such edits , as we can , considering that they change the GAN parameters themselves . ( vi ) Finally , another line of research targets primarily very high-level image and photo stylization and global appearance modifications [ 37 , 53 , 54 , 55 , 52 , 56 , 46 , 57 , 41 ] . Generally , most works only do relatively high-level and not the detailed , high-precision editing , which EditGAN targets . Hence , we consider EditGAN as complementary to this body of work . GANs and Latent Space Image Embedding . EditGAN builds on top of DatasetGAN [ 1 ] and SemanticGAN [ 2 ] , which proposed to jointly model images and their semantic segmentations using shared latent codes . However , these works leveraged this model design only for semi-supervised learning , not for editing . EditGAN also relies on an encoder , together with optimization , to embed new images to be edited into the GAN ’ s latent space . This task in itself has been studied extensively in different contexts before , and we are building on these works . Previous papers studied encoder-based methods [ 58 , 59 , 60 , 61 , 62 ] , used primarily optimization-based techniques [ 63 , 64 , 65 , 66 , 67 , 68 , 69 , 26 ] , and developed hybrid approaches [ 63 , 24 , 25 , 70 , 71 ] . Finally , a concurrent paper [ 72 ] shares similarities with DatasetGAN [ 1 ] , on which our method builds , and explores an editing approach related to our EditGAN as one of its applications . However , our editing approach is methodologically different and leverages editing vectors , and also demonstrates significantly more diverse and stronger experimental results . Furthermore , [ 73 ] shares some highlevel ideas with EditGAN ; however , it leverages the CLIP [ 74 ] model and targets text-driven editing . 3 High-Precision Semantic Image Editing with EditGAN . 3.1 Background . EditGAN ’ s image generation component is StyleGAN2 [ 7 , 8 ] , currently the state-of-the-art GAN for image synthesis . The StyleGAN2 generator maps latent codes z ∈ Z , drawn from a multivariate Normal distribution , into realistic images . A latent code z is first transformed into an intermediate code w ∈ W by a non-linear mapping function and then further transformed into K + 1 vectors , w0 , ... , wK , through learned affine transformations . These transformed latent codes are fed into synthesis blocks , whose outputs are deep feature maps . Deep generative models such as StyleGAN2 , which are trained to synthesize highly realistic images , acquire a semantic understanding of the modeled images in their high-dimensional feature space . Recently , DatasetGAN [ 1 ] and SemanticGAN [ 2 ] built on this insight to learn a joint distribution p ( x , y ) over images x and pixel-wise semantic segmentation labels y , while requiring only a handful of labeled examples . EditGAN utilizes this joint distribution p ( x , y ) to perform high-precision semantic image editing of real and synthesized images . Both methods [ 1 , 2 ] model p ( x , y ) by adding an additional segmentation branch to the image generator , which is a pre-trained StyleGAN [ 1 ] . We follow DatasetGAN [ 1 ] , which applies a simple three-layer multi-layer perceptron classifier on the layer-wise concatenated and appropriately upsampled feature maps . This classifier operates on the concatenated feature maps in a per-pixel fashion and predicts the segmentation label of each pixel .

Summary: This submission tackles the problem of semantic edition of images. The proposed method builds heavily on DatasetGAN.  To edit an image x, the method first finds the corresponding w in the GAN latent space (using a trained encoder) and then optimizes it so that the semantic branch output aligns to a target label map. The process effectively produces a modified image that retains the characteristics of the original image but with a semantic layout given by the target map. The difference between the initial w and the optimized w provides an edit vector, that can be applied to other similar images to perform similar edits in real-time. If needed, some further optimization can still be performed to remove edition artifacts.