Deep Single Image Manipulation

Image manipulation has attracted much research over the years due to the popularity and commercial importance of the task . In recent years , deep neural network methods have been proposed for many image manipulation tasks . A major issue with deep methods is the need to train on large amounts of data from the same distribution as the target image , whereas collecting datasets encompassing the entire distribution of images is impossible . In this paper , we demonstrate that simply training a conditional adversarial generator on the single target image is sufficient for performing complex image manipulations . We find that the key for enabling single image training is extensive augmentation of the input image and provide a novel augmentation method . Our network learns to map between a primitive representation of the image ( e.g . edges and segmentation ) to the image itself . At manipulation time , our generator allows for making general image changes by modifying the primitive input representation and mapping it through the network . We extensively evaluate our method and find that it provides remarkable performance . 1 INTRODUCTION . Images capture a scene at a specific point in time . Viewers often wish the scene had been different e.g . that objects were arranged differently . Due to the popularity of this task , it has been the focus of much research and also of many companies and products e.g . Instagram and Photoshop . Deep learning methods have significantly boosted performance of image manipulation methods for which large training datasets can be obtained e.g . super-resolution or facial inpainting . User-captured photographs follow a long tailed distribution . Some classes of photographs are very common e.g . faces or cars . On the other hand a large proportion of photographs capture a rare object class or configuration . Training deep learning methods that capture the entire distribution images can be very hard , particularly for generative models that are slow and tricky to train . Training models on just the target image is emerging as an alternative to training deep models on large image datasets . Although this is counter-intuitive as deep learning methods typically require many training samples , single-image methods have recently demonstrated some promising results . In this paper , we introduce a novel method for training deep conditional generative models from a single image . The objective differs from popular single-image methods e.g . Deep Image Prior and SinGAN that focus on unconditional image manipulation . The training image is first represented with a primitive representation , which can be unsupervised ( an edge map , unsupervised segmentation ) , supervised ( segmentation map , landmarks ) or a combination of both . We use a standard adversarial conditional image mapping network to learn to map between the primitive representation and the image . In order to extend the training set ( which simply consists of a single image ) , we perform extensive augmentations . The choice of augmentation method makes a significant difference to the method ’ s performance . We find the crop-and-flip augmentations typically used in conditional image generation are insufficient for providing a sufficiently rich training distribution . We propose to use a thin-plate-spline ( TPS ) augmentation method and show that it is key to the success of our method . After training , we are able to perform challenging image manipulation tasks by modifying the primitive representation . Our method is evaluated extensively and displays remarkable results . Our contributions in this paper : 1 . A general purpose approach for training conditional generators from a single image . 2 . Proposing a TPS-based augmentation for conditional image generation , and demonstrating its importance for single image training . 3 . A primitive representation allowing concurrent low and high-level image editing . 4 . Extensive evaluations showing remarkable visual performance , and the introduction of a new benchmark enabling quantitative evaluation of single image manipulation . 2 RELATED WORK . Classical image manipulation methods : Image manipulation has attracted research for decades from the image processing , computational photography and graphics communities . It would not be possible to survey the scope of this corpus of work in this paper . We refer the reader to the book by Szeliski ( 2010 ) for an extensive survey , and to the Photoshop software for a practical collection of image processing methods . A few notable image manipulation techniques include : Poisson Image Editing ( Pérez et al. , 2003 ) , Seam Carving ( Avidan & Shamir , 2007 ) , PatchMatch ( Barnes et al . ) and ShiftMap ( Pritch et al. , 2007 ) . Learning a high-resolution parametric function between a primitive image representation and a photo-realistic image was very challenging for pre-deep learning methods . Deep conditional generative models : Image-to-image translation maps images from a source domain to a target domain , while preserving the semantic and geometric content of the input images . Most image-to-image translation methods use Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) that are used in two main scenarios : i ) unsupervised image translation between domains ( Zhu et al. , 2017a ; Kim et al. , 2017 ; Liu et al. , 2017 ; Choi et al. , 2018 ) ii ) serving as a perceptual image loss function ( Isola et al. , 2017 ; Wang et al. , 2017 ; Zhu et al. , 2017b ) . Existing methods for image-to-image translation require many labeled image pairs . Several methods e.g Dekel et al . ( 2017 ) , are carefully designed for image manipulation , however they require large datasets which are mainly available for faces or interiors and can not be applied to the long-tail of images . Non-standard augmentations : Conditional generation models typically use crop and flip augmentations . Classification models also use chromatic and noise augmentation . Recently , methods have been devised for learning augmentation for classification tasks e.g . AutoAugment ( Cubuk et al. , 2018 ) . Mounsaveng et al . ( 2019 ) learned warping fields for augmenting classification networks . Thin-plate-spline transformation have been used in the medical domain e.g . Tang et al . ( 2019 ) , but they are used for training on large datasets rather than a single sample . Zhao et al . ( 2019 ) learned augmentations for training segmentation networks from a single annotated 3D medical scan ( using a technique similar to Kanazawa et al . ( 2016 ) ) however they require a large unlabeled dataset of similar scans which is not available in our setting . TPS has also been used as a way of parametrizing warps for learning dense correspondences between images e.g . Han et al . ( 2018 ) and Lee et al . ( 2020 ) . Learning from a single image : Although most deep learning works use large datasets , seminal works showed that single image training is effective in some settings . Asano et al . ( 2019 ) showed that a single image can be used to learn deep features . Limited work has been done on training image generators from a single-image . Deep Image Prior ( Ulyanov et al. , 2018 ) , retargeting ( Shocher et al. , 2018a ) and super-resolution ( Shocher et al. , 2018b ) from a single image . Recently , the seminal work of Shaham et al . ( 2019 ) , presented a general approach for single image generative model training . However these methods are mainly focused on unconditional image manipulations ( or very limited conditional manipulation ) . Our method is focused on conditional manipulations and is able to affect significantly more elaborate changes to images including to large objects in the scene . 3 DEEPSIM : LEARNING CONDITIONAL GENERATORS FROM A SINGLE IMAGE . We propose DeepSIM , a conditional generative adversarial network ( cGAN ) for learning to map from a primitive representation ( e.g . edges , segmentation ) to the image . The approach has several objectives : i ) single image training ii ) fidelity - the output should reflect the primitive representation iii ) appearance - the output image should appear to come from the same distribution as the training image . We present a novel augmentation method that allows standard cGAN architectures to achieve these objectives . The particular type of augmentations used is critical as they provide a prior over the data . Having a good prior is important as the training set consists of merely a single image . 3.1 MODEL : . Our model design follows standard practice for cGAN models ( particularly Pix2PixHD ( Wang et al. , 2017 ) ) . Let us denote our training image pair ( x , y ) where y ∈ Rdx×dy×3 is the input image ( dx and dy are the number of rows and columns ) and x ∈ Rdx×dy×dp is the corresponding image primitive ( dp is the number of channels in the image primitive ) . We learn a generator network G : Rdx×dy×dp → Rdx×dy×3 , which learns to map input image primitive x to the generated image G ( x ) . The fidelity of the result is measured using the VGG perceptual loss ` perc : ( Rdx×dy×3 , Rdx×dy×3 ) → R ( Johnson et al. , 2016 ) , which compares the differences between two images using a set of activations extracted from each image using a VGG network pre-trained on the ImageNet dataset ( we follow the implementation in Wang et al . ( 2017 ) ) . We therefore write the reconstruction loss ` rec : ` rec ( x , y ; G ) = ` perc ( G ( x ) , y ) ( 1 ) Conditional GAN loss : Following standard practice , we add an adversarial loss which measures the ability of a discriminator to differentiate between the ( primitive , generated image ) pair ( x , G ( x ) ) and the ( primitive , true image ) pair ( x , y ) . The conditional discriminatorD : ( Rdx×dy×dp , Rdx×dy×3 ) → [ 0 , 1 ] is implemented using a deep classifier which maps a pair of primitive and corresponding image into the probability of the two being a groundtruth primitive-image pair . D is trained adversarially against the generator network G. The loss of the discriminator ( ` adv ) is : ` adv ( x , y ; D , G ) = log ( D ( x , y ) ) + log ( 1−D ( x , G ( x ) ) ) ( 2 ) The combined loss ` total is the sum of the reconstruction and adversarial losses , weighted by a constant α : ` total ( x , y ; D , G ) = ` rec ( x , y ; D ) + α · ` adv ( x , y ; D , G ) ( 3 ) 3.2 AUGMENTATIONS : . When large datasets exist , finding the generator G and conditional discriminator D that optimize ` total under the empirical data distribution can result in a strong generator G. However , as we only have a single image pair ( x , y ) , this formulation severely overfits . This has the negative consequence of G not being able to generalize to new primitive inputs . In order to generalize to new primitive images , the size of the training dataset needs to be artificially increased so as to cover the range of expected primitives . Conditional generative models typically use simple crop and shift augmentations . We will later show ( Sec . 4 ) that this simple augmentation strategy however will not generalize to primitive images with non-trivial changes . We first construct a non-smooth warp t by selecting a set of pixel locations ( i , j ) and perturbing them by a uniformly distributed shifts ( di , dj ) ∼ ( U ( −dxL , dx L ) , U ( − dy L , dy L ) ) s.t . t ( i , j ) = ( i + di , j + dj ) ( we use L = 10 ) . In order to have a simple transformation , we do not do this for every pixel but only for a set of 4 equispaced pixels along each axis of the image . We learn a warp f : ( Rdx , Rdy ) → ( Rdx , Rdy ) using a thinplate-spline ( TPS ) to smooth the random transformation t into a more realistic warp ( i′ , j′ ) = f ( i , j ) . f is parameterized using a thin-plate spline ( see Duchon ( 1977 ) for more details ) . The TPS loss LTPS consists of two terms , a loss that encourages f to be similar to t while the other encourages smoothness : min ω LTPS = ∑ i , j ‖t ( i , j ) − f ( i , j ) ‖2 + λ ∫ ∫ fxx + fyy + 2fxydxdy ( 4 ) Where fxx , fxy , fyy denote the second order partial derivatives of f . λ is a regularization constant determining the smoothness of the warp . The optimization over the warp f can be performed very efficiently e.g . Donato & Belongie ( 2002 ) . We denote the distribution of random TPS that can be generated using the above procedure as Ω .

Summary: This work proposes a method to design conditional generative models based on a single image. In particular, while some recent models have enabled one to sample (unconditionally) images from a generative model learned from a single image (like SinGAN), this work explores a way of conditioning the generation on a primitive, which can be user-specified. As a result, one can produce realistic modifications to a given image by modifying - or sketching - some primitive.