Deep Single Image Manipulation

Image manipulation has attracted much research over the years due to the popularity and commercial importance of the task . In recent years , deep neural network methods have been proposed for many image manipulation tasks . A major issue with deep methods is the need to train on large amounts of data from the same distribution as the target image , whereas collecting datasets encompassing the entire distribution of images is impossible . In this paper , we demonstrate that simply training a conditional adversarial generator on the single target image is sufficient for performing complex image manipulations . We find that the key for enabling single image training is extensive augmentation of the input image and provide a novel augmentation method . Our network learns to map between a primitive representation of the image ( e.g . edges and segmentation ) to the image itself . At manipulation time , our generator allows for making general image changes by modifying the primitive input representation and mapping it through the network . We extensively evaluate our method and find that it provides remarkable performance . 1 INTRODUCTION . Images capture a scene at a specific point in time . Viewers often wish the scene had been different e.g . that objects were arranged differently . Due to the popularity of this task , it has been the focus of much research and also of many companies and products e.g . Instagram and Photoshop . Deep learning methods have significantly boosted performance of image manipulation methods for which large training datasets can be obtained e.g . super-resolution or facial inpainting . User-captured photographs follow a long tailed distribution . Some classes of photographs are very common e.g . faces or cars . On the other hand a large proportion of photographs capture a rare object class or configuration . Training deep learning methods that capture the entire distribution images can be very hard , particularly for generative models that are slow and tricky to train . Training models on just the target image is emerging as an alternative to training deep models on large image datasets . Although this is counter-intuitive as deep learning methods typically require many training samples , single-image methods have recently demonstrated some promising results . In this paper , we introduce a novel method for training deep conditional generative models from a single image . The objective differs from popular single-image methods e.g . Deep Image Prior and SinGAN that focus on unconditional image manipulation . The training image is first represented with a primitive representation , which can be unsupervised ( an edge map , unsupervised segmentation ) , supervised ( segmentation map , landmarks ) or a combination of both . We use a standard adversarial conditional image mapping network to learn to map between the primitive representation and the image . In order to extend the training set ( which simply consists of a single image ) , we perform extensive augmentations . The choice of augmentation method makes a significant difference to the method ’ s performance . We find the crop-and-flip augmentations typically used in conditional image generation are insufficient for providing a sufficiently rich training distribution . We propose to use a thin-plate-spline ( TPS ) augmentation method and show that it is key to the success of our method . After training , we are able to perform challenging image manipulation tasks by modifying the primitive representation . Our method is evaluated extensively and displays remarkable results . Our contributions in this paper : 1 . A general purpose approach for training conditional generators from a single image . 2 . Proposing a TPS-based augmentation for conditional image generation , and demonstrating its importance for single image training . 3 . A primitive representation allowing concurrent low and high-level image editing . 4 . Extensive evaluations showing remarkable visual performance , and the introduction of a new benchmark enabling quantitative evaluation of single image manipulation . 2 RELATED WORK . Classical image manipulation methods : Image manipulation has attracted research for decades from the image processing , computational photography and graphics communities . It would not be possible to survey the scope of this corpus of work in this paper . We refer the reader to the book by Szeliski ( 2010 ) for an extensive survey , and to the Photoshop software for a practical collection of image processing methods . A few notable image manipulation techniques include : Poisson Image Editing ( Pérez et al. , 2003 ) , Seam Carving ( Avidan & Shamir , 2007 ) , PatchMatch ( Barnes et al . ) and ShiftMap ( Pritch et al. , 2007 ) . Learning a high-resolution parametric function between a primitive image representation and a photo-realistic image was very challenging for pre-deep learning methods . Deep conditional generative models : Image-to-image translation maps images from a source domain to a target domain , while preserving the semantic and geometric content of the input images . Most image-to-image translation methods use Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) that are used in two main scenarios : i ) unsupervised image translation between domains ( Zhu et al. , 2017a ; Kim et al. , 2017 ; Liu et al. , 2017 ; Choi et al. , 2018 ) ii ) serving as a perceptual image loss function ( Isola et al. , 2017 ; Wang et al. , 2017 ; Zhu et al. , 2017b ) . Existing methods for image-to-image translation require many labeled image pairs . Several methods e.g Dekel et al . ( 2017 ) , are carefully designed for image manipulation , however they require large datasets which are mainly available for faces or interiors and can not be applied to the long-tail of images . Non-standard augmentations : Conditional generation models typically use crop and flip augmentations . Classification models also use chromatic and noise augmentation . Recently , methods have been devised for learning augmentation for classification tasks e.g . AutoAugment ( Cubuk et al. , 2018 ) . Mounsaveng et al . ( 2019 ) learned warping fields for augmenting classification networks . Thin-plate-spline transformation have been used in the medical domain e.g . Tang et al . ( 2019 ) , but they are used for training on large datasets rather than a single sample . Zhao et al . ( 2019 ) learned augmentations for training segmentation networks from a single annotated 3D medical scan ( using a technique similar to Kanazawa et al . ( 2016 ) ) however they require a large unlabeled dataset of similar scans which is not available in our setting . TPS has also been used as a way of parametrizing warps for learning dense correspondences between images e.g . Han et al . ( 2018 ) and Lee et al . ( 2020 ) . Learning from a single image : Although most deep learning works use large datasets , seminal works showed that single image training is effective in some settings . Asano et al . ( 2019 ) showed that a single image can be used to learn deep features . Limited work has been done on training image generators from a single-image . Deep Image Prior ( Ulyanov et al. , 2018 ) , retargeting ( Shocher et al. , 2018a ) and super-resolution ( Shocher et al. , 2018b ) from a single image . Recently , the seminal work of Shaham et al . ( 2019 ) , presented a general approach for single image generative model training . However these methods are mainly focused on unconditional image manipulations ( or very limited conditional manipulation ) . Our method is focused on conditional manipulations and is able to affect significantly more elaborate changes to images including to large objects in the scene . 3 DEEPSIM : LEARNING CONDITIONAL GENERATORS FROM A SINGLE IMAGE . We propose DeepSIM , a conditional generative adversarial network ( cGAN ) for learning to map from a primitive representation ( e.g . edges , segmentation ) to the image . The approach has several objectives : i ) single image training ii ) fidelity - the output should reflect the primitive representation iii ) appearance - the output image should appear to come from the same distribution as the training image . We present a novel augmentation method that allows standard cGAN architectures to achieve these objectives . The particular type of augmentations used is critical as they provide a prior over the data . Having a good prior is important as the training set consists of merely a single image . 3.1 MODEL : . Our model design follows standard practice for cGAN models ( particularly Pix2PixHD ( Wang et al. , 2017 ) ) . Let us denote our training image pair ( x , y ) where y ∈ Rdx×dy×3 is the input image ( dx and dy are the number of rows and columns ) and x ∈ Rdx×dy×dp is the corresponding image primitive ( dp is the number of channels in the image primitive ) . We learn a generator network G : Rdx×dy×dp → Rdx×dy×3 , which learns to map input image primitive x to the generated image G ( x ) . The fidelity of the result is measured using the VGG perceptual loss ` perc : ( Rdx×dy×3 , Rdx×dy×3 ) → R ( Johnson et al. , 2016 ) , which compares the differences between two images using a set of activations extracted from each image using a VGG network pre-trained on the ImageNet dataset ( we follow the implementation in Wang et al . ( 2017 ) ) . We therefore write the reconstruction loss ` rec : ` rec ( x , y ; G ) = ` perc ( G ( x ) , y ) ( 1 ) Conditional GAN loss : Following standard practice , we add an adversarial loss which measures the ability of a discriminator to differentiate between the ( primitive , generated image ) pair ( x , G ( x ) ) and the ( primitive , true image ) pair ( x , y ) . The conditional discriminatorD : ( Rdx×dy×dp , Rdx×dy×3 ) → [ 0 , 1 ] is implemented using a deep classifier which maps a pair of primitive and corresponding image into the probability of the two being a groundtruth primitive-image pair . D is trained adversarially against the generator network G. The loss of the discriminator ( ` adv ) is : ` adv ( x , y ; D , G ) = log ( D ( x , y ) ) + log ( 1−D ( x , G ( x ) ) ) ( 2 ) The combined loss ` total is the sum of the reconstruction and adversarial losses , weighted by a constant α : ` total ( x , y ; D , G ) = ` rec ( x , y ; D ) + α · ` adv ( x , y ; D , G ) ( 3 ) 3.2 AUGMENTATIONS : . When large datasets exist , finding the generator G and conditional discriminator D that optimize ` total under the empirical data distribution can result in a strong generator G. However , as we only have a single image pair ( x , y ) , this formulation severely overfits . This has the negative consequence of G not being able to generalize to new primitive inputs . In order to generalize to new primitive images , the size of the training dataset needs to be artificially increased so as to cover the range of expected primitives . Conditional generative models typically use simple crop and shift augmentations . We will later show ( Sec . 4 ) that this simple augmentation strategy however will not generalize to primitive images with non-trivial changes . We first construct a non-smooth warp t by selecting a set of pixel locations ( i , j ) and perturbing them by a uniformly distributed shifts ( di , dj ) ∼ ( U ( −dxL , dx L ) , U ( − dy L , dy L ) ) s.t . t ( i , j ) = ( i + di , j + dj ) ( we use L = 10 ) . In order to have a simple transformation , we do not do this for every pixel but only for a set of 4 equispaced pixels along each axis of the image . We learn a warp f : ( Rdx , Rdy ) → ( Rdx , Rdy ) using a thinplate-spline ( TPS ) to smooth the random transformation t into a more realistic warp ( i′ , j′ ) = f ( i , j ) . f is parameterized using a thin-plate spline ( see Duchon ( 1977 ) for more details ) . The TPS loss LTPS consists of two terms , a loss that encourages f to be similar to t while the other encourages smoothness : min ω LTPS = ∑ i , j ‖t ( i , j ) − f ( i , j ) ‖2 + λ ∫ ∫ fxx + fyy + 2fxydxdy ( 4 ) Where fxx , fxy , fyy denote the second order partial derivatives of f . λ is a regularization constant determining the smoothness of the warp . The optimization over the warp f can be performed very efficiently e.g . Donato & Belongie ( 2002 ) . We denote the distribution of random TPS that can be generated using the above procedure as Ω .

Summary: This paper provides an augmentation method to enable single image training. The network learns to map between a primitive representation of the image (e.g. edges and segmentation) to the image itself. During manipulation, the generator allows for making general image changes by modifying the primitive input representation and mapping it through the network.