Deep Single Image Manipulation
Authors: Yael Vinker
Eliahu Horwitz
Nir Zabari
Yedid Hoshen
Venue:  (2021)
Keywords: image translation
single image
conditional generation
image manipulation

Abstract: Image manipulation has attracted much research over the years due to the popularity and commercial importance of the task. In recent years, deep neural network methods have been proposed for many image manipulation tasks. A major issue with deep methods is the need to train on large amounts of data from the same distribution as the target image, whereas collecting datasets encompassing the entire long-tail of images is impossible. In this paper, we demonstrate that simply training a conditional adversarial generator on the single target image is sufficient for performing complex image manipulations. We find that the key for enabling single image training is extensive augmentation of the input image and provide a novel augmentation method. Our network learns to map between a primitive representation of the image (e.g. edges and segmentation) to the image itself. At manipulation time, our generator allows for making general image changes by modifying the primitive input representation and mapping it through the network. We extensively evaluate our method and find that it provides remarkable performance. 

Under review as a conference paper at ICLR 2021
DEEP SINGLE IMAGE MANIPULATION
Anonymous authors Paper under double-blind review
ABSTRACT
Image manipulation has attracted much research over the years due to the popularity and commercial importance of the task. In recent years, deep neural network methods have been proposed for many image manipulation tasks. A major issue with deep methods is the need to train on large amounts of data from the same distribution the target image, whereas collecting datasets encompassing the entire distribution of images is impossible. In this paper, we demonstrate that simply training a conditional adversarial generator on the single target image is sufficient for performing complex image manipulations. We find that the key for enabling single image training is extensive augmentation of the input image and provide novel augmentation method. Our network learns to map between a primitive representation of the image (e.g. edges and segmentation) to the image itself. At manipulation time, our generator allows for making general image changes by modifying the primitive input representation and mapping it through the network. We extensively evaluate our method and find that it provides remarkable performance.
INTRODUCTION
Images capture a scene at a specific point in time. Viewers often wish the scene had been different e.g. that objects were arranged differently. Due the popularity of this task, it has been the focus of much research and also of many companies and products e.g. Instagram and Photoshop. Deep learning methods have significantly boosted performance of image manipulation methods for which large training datasets can be obtained e.g. super-resolution or facial inpainting. User-captured photographs follow long tailed distribution. Some classes of photographs are very common e.g. faces or cars. On the other hand a large proportion of photographs capture rare object class or configuration. Training deep learning methods that capture the entire distribution images can be very hard, particularly for generative models that are slow and tricky to train. Training models on just the target image is emerging as an alternative to training deep models on large image datasets. Although this is counter-intuitive as deep learning methods typically require many training samples, single-image methods have recently demonstrated some promising results.
In this paper, we introduce a novel method for training deep conditional generative models from single image. The objective differs from popular single-image methods c.g. Deep Image Prior and SinGAN that focus unconditional image manipulation. The training image is first represented with a primitive representation, which can be unsupervised (an edge map, unsupervised segmentation), supervised (segmentation map, landmarks) or combination of both. We use standard adversarial conditional image mapping network to learn to map between the primitive representation and the image. In order extend the training set (which simply consists of single image), we perform extensive augmentations. The choice of augmentation method makes a significant difference to the method's performance. We find the crop-and-ffi augmentations typically used in conditional image generation are insufficient for providing sufficiently rich training distribution. We propose to use a thin-plate- plate-spline (TPS) augmentation method and show that key to the success of our method. After training, we are able to perform challenging image manipulation tasks by modifying the primitive representation. Our method is evaluated extensively and displays remarkable results.
Our contributions in this paper:
1. A general purpose approach for training conditional generators from single image.

---

Under review as a conference paper at ICLR 2021
Training Image Pair
Input
Output
edge to image
seg. to image
seg. to image
Figure I: Results produced by our model. The model was trained on a single training pair (first and the second columns). The third column shows the inputs trained model inference time, First row- (left) lifting the nose, (right) flipping the eyehrows. Second row- (left) adding wheel, (right) conversion a sports car. Third row modifying the shape of the starfish.
2. Proposing TPS-based augmentation for conditional image generation, and demonstrating its importance for single image training.
3. A primitive representation allowing concurrent low and high-level image editing.
4. Extensive evaluations showing remarkable visual performance, and the introduction of new benchmark enabling quantitative evaluation of single image manipulation.
2 RELATED WORK
Classical image manipulation methods: Image manipulation has attracted research for decades from the image processing, computational photography and graphics communities. It would not be possible to survey the scope of this corpus of work in this paper. We refer the reader to the book by Szeliski 2010) for extensive survey, and the Photoshop software for practical collection of image processing methods. A few notable image manipulation techniques include: Poisson Image Editing (Pérez al. (200). Seam Carving (Avidan & Shamir 2007). PatchMatch (Barnes al.) and ShiftMap (Pritch et al.) 2007). Learning a high-resolution parametric function between a primitive image representation and photo-realistic image was very challenging for pre-deep learning methods.
Deep conditional generative models: Image-to-image translation maps images from source domain target domain. while preserving the semantic and geometric content of the input images. Most image-to-image translation methods use Generative Adversarial Networks (GANs) (Goodfellow al. 2014) that are used two main scenarios: i) unsupervised image translation between domains (Zhu et al. 2017a Kim et al. 2017 Liu et al.) 2017;; Choi et al.] 2018) ii) serving perceptual image loss function (Isola et al. 2017 Wang et al. 2017 Zhu et al. 2017b). Existing methods for image-to-image translation require many labeled image pairs. Several methods e.g Dekel et al. (2017). are carefully designed for image manipulation, however they require large datasets which are mainly available for faces or interiors and cannot be applied to the long-tail tail of images.
Non-sttee standard augmentations: Conditional generation models typically use crop and flip augmen- tations. Classification models also use chromatic and noise augmentation. Recently, methods have
2

---

Under review as a conference paper at ICLR 2021
been devised for learning augmentation for classification tasks e.g. AutoAugment (Cubuk al. 2018). Mounsaveng et al. (2019) learned warping fields for augmenting classification networks. Thin-plate-splinee transformation have been used in the medical domain e.g. Tang et al. (2019). but they are used for training on large datasets rather than a single sample. /Zhao et al. (2019) learned augmentations for training segmentation networks from a single annotated 3D medical scan (using technique similar to Kanazawa et al. (2016)) however they require a large unlabeled dataset of similar scans which is not available in our setting. TPS has also been used as a way of parametrizing warps for learning dense correspondences between images e.g. Han et al. (2018) and Lee et al. (2020).
Learning from a single image: Although most deep learning works use large datasets, seminal works showed that single image training is effective in some settings. |Asano et al.] showed that a single image can be used to learn deep features. Limited work has been done on training image generators from single-image. Deep Image Prior (Ulyanov et al. (018), retargeting (Shocher et al., 2018a) and super- resolution (Shocher et al., 201 from single image. Recently, the seminal work of Shaham et al. (2019), presented general approach for single image generative model training. However these methods are mainly focused on unconditional image manipulations (or very limited conditional manipulation). Our method is focused on conditional manipulations and is able to affect significantly more elaborate changes to images including to large objects in the scene.
DEEPSIM: LEARNING CONDITIONAL GENERATORS FROM A SINGLE IMAGE
We propose DeepSIM, a conditional generative adversarial network (cGAN) for learning to map from primitive representation (e.g. edges, segmentation) the image. The approach has several objectives: i) single image training ii) fidelity- the output should reflect the primitive representation iii) appearance the output image should appear to come from the same distribution as the training image. We present novel augmentation method that allows standard cGAN architectures to achieve these objectives. The particular type of augmentations used critical as they provide a prior over the data. Having good prior is important as the training set consists of merely a single image.
3.1 MODEL:
Our model design follows standard practice for cGAN models (particularly Pix2PixHD (Wang et al. 2017h). Let us denote our training image pair (x,, y) where y E [d,, is the input image (d,, and dy are the number of rows and columns) and x E IR xd, Xdy is the corresponding image primitive (dp is the number of channels in the image primitive). We learn generator network G : Rdd xd, xdp + Xd,xxx which learns to map input image primitive
the generated image G(z). The fidelity
of the result is measured using the VGG perceptual loss Epere (R xd, x3 Rd xd, x3) + R (Johnson et al. 2016), which compares the differences between two images using a set of activations extracted from each image using VGG network pre-trained on the ImageNet dataset (we follow the implementation in Wang et al. (2017)). We therefore write the reconstruction loss Erea:
(x y: G) = (pere(((z). v)
(1)
Conditional GAN loss: Following standard practice, we add an adversarial loss which measures the ability of discriminator to differentiate between the (primitive, generated image) pair (z. G(x)) and the (primitive, true image) pair (z, 3). The conditional discriminator D : [Rd.xd,xd, 1dd. xdyxx) + [0, 1 is implemented using deep classifier which maps pair of primitive and corresponding image into the probability of the two being a groundtruth primitive-iman image pair. D is trained adversarially against the generator network G. The loss of the discriminator (Lade) is:
lado y; D. = log ((x,)) + log(1 - D(r, G(=)))
(2)
The combined loss trotal is the sum of the reconstruction and adversarial losses, weighted by a constant
Exotat (I. y5 D.G) bree y5 D) tado (x, y D,
(3)
3

---

Under review as a conference paper at ICLR 2021
Primitive
Image
Primitive after TPS
Image after TPS
Table A source image and a primitive representation under a random TPS warp. Also see App. H
3.2 AUGMENTATIONS:
When large datasets exist, finding the generator G' and conditional discriminator D that optimize lotal under the empirical data distribution can result in a strong generator G. However, as we only have single image pair yI, this formulation severely overfits. This has the negative consequence of G not being able to generalize to new primitive inputs. In order to generalize to new primitive images, the size of the training dataset needs to be artificially increased as cover the range of expected primitives. Conditional generative models typically use simple crop and shift augmentations. We will later show (Sec. 4 that this simple augmentation strategy however will not generalize to primitive images with non-trivial changes. We first construct a non-smooth warp t by selecting a set of pixel locations (i,j) and perturbing them by uniformly distributed shifts (de, dz) ~ (1(-444),00 have simple transformation, T.
dy dy s.t. (i..j) = (i + d;, + dj) (we use = 10). In order to do not do this for every pixel but only for set of 4 equispaced pixels along each axis of the image. We learn a warp f (IRd,, 1,d) + plate-spline (TPS) smooth the random transformation into more realistic warp (i', = f(i f is parameterized using a thin-plate spline (see Duchon (977) for more details). The TPS loss
(12), using a thin-
LTps consists of two terms, a loss that encourages f to be similar to while the other encourages smoothness:
f fuu fax 2f
min LTPS
(4)
1.j
Where fee: fay. foy denote the second order partial derivatives of f. A is a regularization constant determining the smoothness of the warp. The optimization over the warp f can be performed very efficiently e.g. [Donato & Belongie (2002). We denote the distribution of random TPS that can be generated using the above procedure as 12.
3.3 OPTIMIZATION:
At training time, we sample random warps from the distribution Q. We use each random warp ~~ Ω2 to transform both the input primitive x and image y create (where we denote (t,j) = a(i', (') where (i''') = f((, We optimize the generator and discriminator adversarially minimize the expectation of the loss lotall under the empirical distribution of random TPS warps:
training pair (f(z), f()))
D', G' = min max max
(5)
G D
We implemented the conditional GAN using the Pix2PixHD architecture. We kept the same hyperpa- rameters in the official repository except changing the number of iterations 16000.
3.4 PRIMITIVE IMAGES:
In order to be able to edit the image, we condition our generator representation of the image that we call the image primitive. The required properties of the image primitives are being able to precisely specify the required output image and the ease of manipulation by image editor. These two objectives are in conflict, although the most precise representation of the edited image is the edited image itself, this level of manipulation is impossible to achieve by human editor, in fact simplifying this representation is the very motivation for this work. Two standard image primitives used by previous conditional generators the edge representation of the image and the semantic
4

---

Under review as a conference paper at ICLR 2021
Source image
Edges Only
Only
Super Primitive
Figure 2: Results on three different image primitives (the leftmost column shows the source image, then each column demonstrate the result of our model when trained on the specified primitive). We manipulated the image primitives, adding a right eye, changing the point of view and shortening the beak. Our results are presented next to each manipulated primitive. Our SP performed best on high-level changes (e.g. the eye), and low-level changes (e.g. the background).
Input Image Pair
Impur
Ouupun
Figure 3: Results of our SP on challenging image manipulation tasks. left) the primitive-imen image pair used to train our method. center) switching the positions between the two rightmost cars. right) removing the leftmost car and inpainting the background. In both cases our method was able to synthesize very attractive output images. See App. for many more results.
instance/segmentation map of the image. Segmentation maps provide information on the high-level properties of the image, but give less guidance on the fine-details. Edge maps provide the opposite trade-off. To achieve the best of both worlds, we propose a novel primitive, which includes both the edge and segmentation maps combined together (we dub this representation "super primitive"). The advantages of this primitive representation will be shown in Sec. 5 Note that although Pix2Pix- HD proposes using a combination of segmentation maps and the edges of the instance maps, their representation is different from ours. The instance segmentation edges merely provide a method for overcoming the indexing issue present in instance maps, but do not provide the high-resolution of fine details in the edge maps as in our proposed approach. Our editing procedure is illustrated in App.
B
4 EXPERIMENTS
4.1 QUALITATIVE EVALUATION
We present many results of our method in the main paper and the appendix. Fig. I our method is able to generate very high resolution images from single image training. In the top row we are able to perform fine changes to the facial images from edge primitives e.g. raising the nose and flipping the eyebrows. In the second row, show complex shape transformations by using segmentation primitives. Our method was able to add a third wheel to the car and convert its shape into a sports car. This shows the power of the segmentation primitive, enabling major changes the shape using simple operations. On the bottom row, we show that our method can perform free-form changes, completely changing the shape of an image while preserving fine texture.
5

---

Under review as a conference paper at ICLR 2021
Training Image
Input
PiX2RKHD--I
BleydkGAN
Ours
RIS
Figure 4: Edges-to-image results. columns 1, 2 show the edges and images used for training. Column
3 shows the edges used as input inference time. We can see that Pix2PixHD-MI cannot generate the correct shoe there is not enough guidance. BicycleGAN has sufficient guidance but cannot reproduce the correct details. Our approach generates images of high quality and fidelity.
Training Image Pair
Input
SinGAN
Ours
Figure 5: Paint-to-image results. Two leftmost columns show the input paint that was created manually and the input image. The third column shows the modified paint image used as input the trained models, the result by SinGAN was generated using the authors' best practice. Our method generates a novel tree corresponding the segmentation map with high fidelity. Note that SinGAN is not designed for conditional generation, hence the weak results.
In Fig. 4 we demonstrate the advantages of our single image model by comparing to two models that were trained shoes. We can see that Pix2PixHD-MI (Pix2PixHD that was trained on the entire edge2shoes dataset, where "MI" is an acronym for "Multi Image") is unable to capture the correct identity of the shoes as there are multiple possibilities for the appearance of the shoe given the edge image. BicycleGAN is able to take input both the edge map and guidance for the appearance (style) of the required shoe. Although it is able to capture the general colors of the required shoe, it is unable to capture the fine details of the shoes (e.g. shoe laces and buckles). We believe that the loss of information is general disadvantage of training on large datasets, general mapping function becomes less specialized and therefore less accurate on individual images.
large dataset. We present edges-to-in toess-tt-mmage translation results two different
In Fig. 5 we present results paint image to an image of tree, while SinGAN was trained using the authors' best practice. We can see that SinGAN outputs an image which is more similar to the paint than photorealistic image. Our method is able to change the shape of the tree to correspond to the paint while keeping the appearance of the tree and background as in the training image.
paint-to-image task. Our method was trained to map from rough
4.2 QUANTITATIVE EVALUATION
As previous single image generators have mostly operated on unconditional generation, there are no established datasets and metrics to conduct such an evaluation. We propose a new pideo-based benchmark for conditional single image evaluation over datasets spanning different scenes and primitives. A single frame from each video is designated for training, where the network is trained to map the primitive image to the designated training frame. The trained network is then used to map from primitive to image for all the other frames in the video and compute the prediction error using LPIPS (Zhang et al.) 2018) and fidelity using SIFID (Shaham et al. 2019). We use all 16 video segments from the Cityscapes dataset (Cordts et al. 2016) provided by the code in vid2vid (Wang

---

Under review as a conference paper at ICLR 2021
Inpuc
PiXPPiHD-SSA
Ourr
Ground Truth
Training Image Pair
Figure 6: Several sample results from the Cityscapes dataset. We train each model the segmentation-imes imag pair on the left. We then use the models to predict the image, given the segmentation maps (second column from left). Our method shown to perform very well on this lask, generating novel configurations of people not seen in the training image.
Training Image Pair
Input
Pix2PiHD-SSA
Ours
Ground Truth
Figure 7: Comparing Pix2PixHD trained on single image pair with the "flip-and-crop" warp vs.
DeepSIM. Our method outputs an image much more similar to the ground truth. Also see App. D
ctt al., 2018) and Few-shot-vid2vid (Wang et al. 2019). These sequences are labelled sachen-000000 aachen-000015 leftImg8hit. For each sequence, train frame 000000 and test using frames 000001 to 000029. We use the segmentation maps provided as image primitives. We also use the first videos in the public release of the Oxford-BBC Lip Reading Sentences (LRS2) dataset containing videos of different speakers. We extract their edges using a Canny edge detectorCanny (1866). In total, our evaluation set contains 464 Cityscapes frames and 239 LRS2 frames.
A visual evaluation our method to the results of Pix2PixHD-SS a Pix2PixHD model that was trained on single image using random crop-and-flip warps but not TPS. We can observe that our method is able to synthesize very different scene setups from those seen in training, including different numbers and positions of people. Our method outperforms significantly terms of fidelity and quality than Pix2PixHD-SII indicating that our proposed TPS augmentation is critical for single image conditional generation.
few frames from the Cityscapes dataset can be seen in Fig. We compare where "SIA" stands for "Single Image Augmented" c.g.
Quantitative evaluations on Cityscapes and LRS2 are provided Tab. 2 and Tab. 3 We report LPIPS and SIFID for each of the LRS2 sequences and also for the average of 16 Cityscapes videos. We can observe that our method significantly outperforms Pix2PixHD-SIP in all comparisons.
ANALYSIS
Why use Suppe--prinitii((SP)? The choice of different input primitives conveys different information. Segmentations capture high-level aspects the image while edge maps capture the low-level of the image better. Pix2PixHD proposed combining instance and semantic segmentation maps, however, this does not provide low-level details. We propose a new primitive, "super primitive" (SP), that combines their strengths. Fig. 2 compares the three primitives. The edge representation is unable to capture the eye, presumably as it cannot capture its semantic meaning. The segmentation is unable to capture the details in the background regions creating smearing effect. SP is able capture the eye as well as the low-level textures the background region, showing its strong representational present more manipulation results using SP. In the center column, present power. In Fig.
7

---

Under review as a conference paper at ICLR 2021
Table 2: Results for the Cityscapes dataset we report the average the 16 videos.
Metric Baseline: Seg, Crop+Flip
Ours: Seg, TPS
Ours: SP, TPS
LPIPS SIFID
0.342
0.216
0.134
0.292
0.127
0.104
Table 3: Results Pix2PixHD-Sie (crop-and-fij) and our method (TPS) on 5 LRS2 vidcos (both trained on single pair). For each sequence left column: LPIPS, right column: SIFID.
Method
SI(L/S)
S2(L/S)
S3(L/S)
S4(L/S)
S5(L/S)
Pix2PixHD
0.44
0.51
0.47
0.49
0.41
0.5
0.53
0.26
0.46
0.44
Ours no VGG
0.14
0.05
0.26
0.11
0.11
0.07
0.28
0.14
0.19
0.08
Ours
0.12
0.07
0.21
0.12
0.1
0.04
0.22
0.12
0.14
0.06
image reorganization results, where the positions of the rightmost cars were switched. As the objects were not of the same size, some empty image regions were filled using small changes to the edges. more extreme result can seen the rightmost column, the car on the left was removed. This created a large empty image region. By filling in the missing details using edges, our method was able to successfully complete the background. An ablation is presented in App. E showing the superiority of the SP representation. More comparisons are presented in App. D
the cGAN loss necessary? We evaluated removing the cGAN loss used by our method (and Pix2PixHD), and using VGG perceptual loss (the difference between the activations of VGG network of output and groundtruth). The results on Cars are presented in App. F we conclude that for such high-res images the cGAN is better perceptual loss. We experimented with the VGG loss at lower resolutions, the results are reasonable, however the images are not as sharp as the cGAN loss.
Can methods trained on large-scale datasets generalize to infrequently occurring images? We present examples where this is not the case. Fig. [ showed that BicycleGAN did not generalize as well as Pix2PixHD-MI for new (in-distribution) shoes. We show that in the more extreme case, where the image lies further from the source distribution used for training, current methods fails completely. We manually labelled the semantic and instance segmentation maps of the Cars image, and input into a Pix2PixHD pre-trained by the authors on the Cityscapes dataset which contains street scenes of cars, roads and building. The results are presented in Tab. 4 We can see that state-of of-t-------artt method pre-trained on large dataset does not generalize well to out-of nut-of-distributio inputs whereas our method, which trained on a single image can perform well such cases.
Augmentation for single image manipulation: Although we are the first to propose single-image training for manipulation using extensive non-linear augmentations, SinGAN can seen as implicitly being an ingmentation-bsee approach (for unconditional generation). In the first pyramid level it creates a low-resolution high-level random image generator, by mapping noise to low-res images. The later stages can be seen as conditional generator which learns to map low-to high resolution images. Critically, relies on a set of "augmented" input low-res images generated by the first stage GAN. So although this is not explicit in SinGAN, we think it also uses augmentations. On the other hand, Deep Image Prior indeed does not use any form of augmentation.
Input Semantic Seg.
Input Instance Seg.
Output
Ground Truth
Table 4: Running Cityscapes-trainel trained Pix2PixHD on primitive representaation of an out-of- distribution image. The network was not able to generalize well and generated unsatisfactory results.
8

---

Under review as a conference paper at ICLR 2021
Training Image Pair
Output
Figure 8: Failure modes: (left) generating unseen objects eyes of the dog (center) background duplication sea behind the turtle (right) empty space interpolation nose the cat
Runtime: As with other methods that train deep neural networks based on single image, per-image training times are higher than when trained on large datasets. Our runtime a function of the neural architecture
use and the number of iterations. Here specifically, framework which has roughly comparable runtime to SinGAN for the same image size. The precise runtime depends on image size and the number of iterations used. When running all experiments the same hardware (NVIDIA RTX-2080 Ti), smaller images e.g. the "Balloons" image showcased by SinGAN (size: 248X 16) take SinGAN 50 minutes while DeepSIM (ours) takes 63 minutes. Runtime scales with the size of the image, so that large images such as the Cars image (640X320) take SinGAN 195 minutes while ours takes 185 minutes. Although this is not particularly fast, is general characteristic of many deep single image manipulation methods and not a particular issue of DIM. We are optimistic that runtime optimizations in future work can significantly cut the runtime in all such methods, but this is not our focus.
use the Pix2Pix-HD
Learning from single image very challenging and although our method makes encouraging progress, it has failure cases. We highlight three main failure modes (Fig. [): generating unseen objects the network trained on just single pair and not aware of any other objects. When the manipulation requires generation of unseen objects, the network can do so incorrectly. ii) background duplication when adding an object onto new background regions, the network can erroneously copy some background regions that originally surrounded the object, which may appear out-of-context iii) interpolation in empty regions as no guidance is given in empty image regions, the network hallucinates details, sometimes incorrectly. See App. for further analysis.
Failure modes:
6 CONCLUSION
We proposed novel method for training conditional generators from single training image based on thin-plate-spline augmentations. We demonstrated that our method is able to perform complex image manipulation at high-resolution. Single image methods have significant potential, they preserve image fine-detalls to level not typically achieved by previous methods trained on large datasets. One limitation of single-image methods (including ours) is the requirement for training separate network for every image, which can be expensive over a large dataset. Speeding up training single-image generators is an important and promising direction for future work.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi. A critical analysis of self-supervision, or what we can learn from single image. arXiv preprint arXiv: 1904. 13132, 2019.
Shai Avidan and Ariel Shamir. Seam carving for content-awart image resizing. In SIGGRAPH, 2007.
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing.
J Canny. A computational approach to edge-detection. leee transactions on pattern analysis and machine intelligence, 1986.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Wor Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In CVPR, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint (xXxv:/805.09500) 2018.
Tali Dekel, Chuang Gan, Dilip Krishnan, Ce Liu, and William Freeman. Smart, sparse contours to represent and edit images. arXiv preprint 1rXiv:1712.88232 2017.
Gianluca Donato and Serge Belongie. Approximate thin plate spline mappings. In European conference on computer vision, pp. 21-31. Springer, 2002.
Jean Duchon. Splines minimizing rotation-invariant semi-norms in sobolev spaces. In Constructive theory functions of several variables, pp. 85-100. Springer, 1977.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, pp. 2672-2680, 2014.
Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In CVPR, 2018.
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
Angjoo Kanazawa, David W Jacobs, and Manmohan Chandraker. Warpnet: Weakly supervised matching for single-view reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3253-3261, 2016.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jungkwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In ICML, 2017.
Junsoo Lee, Eungyeup Kim, Yunsung Lee, Dongjun Kim, Jachyuk Chang, and Jaegul Choo. Reference-based sketch image colorization using augmented-self reference and dense seman- tic correspondence. In CVPR, 2020.
Ming-Yo Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In NIPS, 2017.
Saypraseuth Mounsaveng, David Vazquez, Ismail Ben Ayed, and Marco Pedersoli. Adversarial learning of general transformations for data augmentation. arXiv preprint arXiv: 2019.
Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing. In SIGGRAPHs, 2003.
10

---

Under review as a conference paper at ICLR 2021
Yael Pritch, Eitam Vevakenaki, and Shmuel Peleg. Shift-map image editing. In /CCV, 2007.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning generative model from single natural image. In Proceedings of the IEEE International Conference on Computer Vision, pp.
2019.
Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irami. Internal distribution matching for natural image retargeting. CoRR, abs/1812.00031, 2018a. URL httt::t/aaattttt org. wwtp://gg//aaaaaa.oooo...o... abs/ 1812
00231
Assaf Shocher, Nadav Cohen, and Michal Irani. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 31 3118-3126, 2018b.
Richard Szeliski. Computer vision: algorithms and applications. Springer Science & Business Media, 2010.
Zhixian Tang, Kun Chen, Mingyuan Pan, Manning Wang, and Zhijian Song. An augmentation strategy for medical image processing based on statistical shape model and 3d thin plate spline for deep learning. IEEE Access,
2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446-9454, 2018.
Ting- Ting-Chun Wang, Ming-Yu Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. CoRR, abs/1711.11585., 2017. httpht
abs
Ting-Chun Chun Wang. Ming- Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Ting-Chun Wang, Ming- Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to-video synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. arXiv preprint arXiv: 1oXiv:1801.03224 2018.
Amy Zhao, Guha Balakrishnan, Fredo Durand, John V Guttag, and Adrian V Dalca. Data augmenta- tion using learned transformations for one-shot medical image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8543-8553, 2019.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017a.
Jun Yan Zhu. Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems, pp. 465-476, 2017b.
A ADDITIONAL RESULTS
We present additional results our method, DeepSIM, using the SP on a range of manipulations on different images.
11

---

Under review as a conference paper at ICLR 2021
Training SP
Training Image
Input SP
Output Image
Changing the Posture of the Squirrel
Training SP
Training Image
Input SP
Output Image
Changing the Shape of the Tail
Removing the Fork
Joining the Hamburger Halves
12

---

Under review as a conference paper at ICLR 2021
Training SP
Training Image
Input SP
Ouiput Image
Making the Beak Longer
Changing the Position of the Wings
Removing the Handcuffs
Removing the Right Hand
13

---

Under review as a conference paper at ICLR 2021
Training SP
Training Image
Input SP
Output Image
Changing the Hat and Making the Face Longer
Making the Body Wider
Moving the Tree
Changing the Shape of the Tree
14

---

Under review as a conference paper at ICLR 2021
Training SP
Training Image
Input SP
Output Image
Changing the Shape of the Lake
Adding More Lakes
Removing the Teeth of the Top Lama
Changing the Shape the Ears of the Top Lama
15

---

Under review as a conference paper at ICLR 2021
Training SP
Training Image
Input SP
Output Image
Adding Stems
Changing the Shape of the Horns
Adding the Left Paw
B A STEP-BY-STEP DEMONSTRATION OF EDITING THE PRIMITIVE
Performing complex manipulations by our method is quite easy. In this figure we present a step-by- step example of editing primitive representation using "Paint". It simply requires sampling the required color and painting over the primitive image. One may also "borrow" edges from other areas of the image fill in empty spaces.
16

---

Under review as a conference paper at ICLR 2021
Step 1: Original Image
Step 2: Paint Heart
Step 3: Copy Edges of Trees
Step 4: Rearrange Edges of Trees
Figure 9: Evaluation of the ability of our network to interpolate across empty space regions. The two leftmost columns show the training image pair, gradually increase the distance between the eyes and nose of the cat, and feed the test images the network, the corresponding output of each test image is shown in the second row. Our method generates attractive interpolations for moderate changes, the performance deteriorates for larger interpolations.
C EMPTY SPACE INTERPOLATION
In this section stress test our method's ability handle regions with little guidance. In this example, the nose of the cat was shifted progressively downwards, forcing the network to interpolate the missing space. We observe the network synthesizes attractive images for moderate empty regions, however, as the empty region gets larger, the network looks for similar regions to fill the newly created void. These regions will often be areas which exhibit low amounts detail in the primitive representation. In our case can notice that for larger shifts, the empty space becomes greener until eventually it inpaints a background patch. We conclude that at certain point, the network fails to learn the spatial relationship among objects in the image (i.e. that the background not be placed on the cat's 's face) and satisfies the given constraint using neighboring information (as was analysed above).
17

---

Under review as a conference paper at ICLR 2021
D VIDEO FRAMES
We present a qualitative comparison between different primitives on two frames from the LRS2 datasets. Although all primitives generate surprisingly good results, given the training on just single image, super-primitive generates cleaner outputs with fewer artifacts.
Training Primitive
Training Image
Input Primitive
Output Image
Ground Truth
Training Primitive
Training Image
Input Primitive
Output Image
Ground Truth
18

---

Under review as a conference paper at ICLR 2021
Training Primitive
Training Image
Input Primitive
Output Image
No
Figure 10: Ablation of SP on the cats dataset: (top) edges-only (center) segmentation-one only (bottom) SP. We can see that edges-only creates wrong associations between objects, segmentation-only fails to generate the fine details correctly (e.g. building), whereas SP achieves strong results.
E ABLATION OF SP FOR THE CARS IMAGE
We present an ablation of the superprimitive (SP) representation for the Cars image. In Fig. 10 we present results for manipulation the Cars image using edge-only, segmentation-onn only and SP. We can see that SP generates attractive artifact free results.
F AN ABLATION OF THE LOSS OBJECTIVE
We compare the results of our method, DeepSIM, using the original cGAN loss in the base Pix2PixHD architecture vs. non-adversarial losses the simple Lı loss and the percetual loss based on the difference of VGG activations. We can see that on this image both non-adversarial losses fail. Note that at lower resolutions non-adversaridl losses do indeed succeed but do not generate results of comparable sharpness of the cGAN loss. Additionally, we performed the experiment with the cGAN but without the VGG perceptual loss, the results are presented below. It can be seen that without the VGG loss, the results are reduced quality and contain grainy artifacts.
Training
Perceptual Loss
L1 Loss
Ours w/o VGG
Ours w/ VGG
19

---

Under review as a conference paper at ICLR 2021
Train edge
Test edge
KNN no TPS
KNN TPS
Train image
Test image
Predicted image
Prediction error
Figure 11: An analysis of the benefits of TPS. We show the kNN distance between patches in the test and train frames with and without TPS augmentations (top-right). We can see that TPS augmentation decreases the kNN distance, in some image regions the decrease is drastic suggesting the patches there can be obtained by deformations of training patches. The kNN-TPS distance appears to be correlated with the regions where the prediction errоr of our method is large. This analysis suggests that by artificially increasing the diversity of patches, single-image methods can generalize better to novel images.
G TPS GENERALIZATION IMPROVEMENT
Let consider the train and test edge-image pairs presented in Fig. 11 We input each edge map through an ImageNet-trained ResNer50 network and computed the activations at the end the third residual block. For each pixel in the activation grid of the test image, we computed the nearest- neighbor (INN) distance to the most similar activation of the train image. We then performed 50 TPS augmentations to the training image, and repeated the INN computation with the training set now containing the activations of the original training image and its 50 augmentation. Let compare the INN distances presented in Fig. IT with and without TPS augmentations. Naturally, the 1NN distance decreased for the TPS-auumented training set due its larger size. More interestingly, we can that several face regions which prior to the augmentations did not have similar patches the input, now have much lower distance (while more significant changes might not be possible describe by TPS). In Fig. II we present the results of our method when trained on the training edge-image pair (shown in the leftmost column) and evaluated
that the prediction error (I, difference between ResNet50 activations of the predicted and the true test image) appears be strongly related the INN distance with TPS- TPS-augmentations. This gives some evidence the hypothesis that the network recalls input-output pairs seen in training. It also gives an explanation for the effectiveness of TPS training, namely increasing the range of input-output pairs thus generalizing to novel images.
the test edge. We
H TPS EXAMPLES
We present several examples of original and TPS augmented images and primitives. We that TPS introduces complex deformations to the samples, allowing much more expressive edits than when using simple "flip and crop" augmentations.
20

---

Under review as a conference paper at ICLR 2021
Original
TPS
TPS
TPS 3
Original
TPS 1
TPS
TPS 3
21


Reviews:
[{"review_id": "qLjE5R6Kc6S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides an augmentation method to enable single image training. The network learns to map between a primitive representation of the image (e.g. edges and segmentation) to the image itself. During manipulation, the generator allows for making general image changes by modifying the primitive input representation and mapping it through the network.\nOn the positive side:\n- The paper proposes an interesting mechanism to train conditional generators from a single image.\n- The proposed super primitive works well for single image manipulation tasks.\n- Some good image editing results are shown in the experiments.\nOn the negative side:\n- The method requires a professional editing ability for editing edges of a super primitive. The generation of primitives also highly depends on the accuracy of semantic segmentation. If the segmentation is done manually, the editing process maybe time-consuming.\n- The technical contribution is limited. It seems that the kernel of the framework is a direct use of cGAN without introducing many new ideas.\n- The training speed is not reported. The reviewer thinks that a fast training process is important for single image manipulation.\n- The qualitative evaluation is not very convincing. It’s better to conduct a user study.", "extra_scores": {}, "extra_text": {}}, {"review_id": "kBm9YAoai6L", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "<Paper Summary>\nThis work proposes a method to design conditional generative models based on a single image. In particular, while some recent models have enabled one to sample (unconditionally) images from a generative model learned from a single image (like SinGAN), this work explores a way of conditioning the generation on a primitive, which can be user-specified. As a result, one can produce realistic modifications to a given image by modifying - or sketching - some primitive.\n\n<Review Summary>\nI like the simplicity and the ingenuity of the approach. This reviewer is not aware of any method that can produce similar results, and as such it represents the state of the art for deep-learning based single-image manipulation. At the same time, clarity (of both writing and technical aspects) could be significantly improved. \n\n<Details>\nStrengths:\n* Novel formulation to train 'single' image generative models\n* Flexible framework\n* Compelling experimental results\n\nWeaknesses:\n* While it is true that this model trains a generative model from a single image, it does so by deploying traditional large-scale learning on many modified versions of a single input image. This is in contrast to other methods (DIP, SinGAN) that train completely on *a single image*. This distinction could be made clearer in the text.\n* Many comments and expressions are too vague, making it difficult to fully understand the approach: In GAN models, the discriminator, $D$, is typically a deep-network model parametrizing a function from the space of images to the reals, say $R^n \\rightarrow [0,1]$, representing the probability that the input comes from the distribution of real images as opposed to from a synthetic generator, $G$. In Eq. (2), however, the discriminator model seems to receive two outputs, $(x,G(x))$, which makes little sense to me. Did the authors perhaps meant to write simply $D(y)$ and $D(G(x))$ in Eq. 2? More broadly, this confusion comes from a lack of a clear definition of the employed functions.\n* Also in Eq. (2), GANs typically have a large collection of training samples and so writing the expectation over the distribution of images $p_{data}$, makes sense. In this case, however, one has only 1 sample. It's true that the authors are artificially creating a distribution around the given sample, so perhaps they could make this more precise and clarify what they refer to as $p_{data}$.\n* It would significantly help the presentation to define the different quantities and spaces used by the authors. For example, consider defining the domain and codomain of the warp $f$, which they employ to generate the augmentation. It is also not totally clear how these transformations are sampled (the $t(i,j)$'s). Later in Sec. 3.4 the authors mention that they randomly sample a new TPS wrap, but sampled how and from what distribution?\n*  There's no comment as to how computational intensive the method is: How many new samples (augmentations) are generated for every image? How long does it take to generate these, and how long does the subsequent training take?\n* \"The long tail of images\" doesn't mean anything to this reviewer. It is clear that the authors intend to refer to images that occur very infrequently in the distribution of real images, but the authors should make this more precise and avoid comments of the sort of \"primitive from the long-tail\", which make no sense.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FY2_muhRcU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a single image-based manipulation method (DeepSIM) using conditional a generative model. The authors addressed this problem by proposing to learn the mapping between a set of primitive representation, which consists of edges and segmentation masks, and an image. They also adopted a thin-plate-splines (TPS) transformation as augmentation which enables the model to robustly manipulate an image by editing primitives.\n\nPros\n-\tThis paper is clearly written and easy to follow.\n-\tThe authors proposed a novel conditional manipulation method based on a single image, which is new in this area.\n-\tDeepSIM is capable of generating the plausible results by manipulating its contents in both a low and a high-level manner, maintaining its realism and fidelity.\n\nCons\n-\tThe authors need to clarify why the VGGNet-based perceptual loss encourages the model to maintain the fidelity. As Johnson et al. [1] argued that this loss is defined as humans’ perceptual difference between the images, it would be helpful to further explain how the model with this loss between G(x) and y better reflects the primitive representations in the generated output than the model without it. Additional explanation and experiments, such as ablation study, would make the paper convincing. \n-\tCompared to the segmentation, the edge map seems to less contribute to the image manipulation. Most of the results are mainly attributed to the segmentation changes, and only slight modification is caused by the change of edge primitive, as shown in Appendix. A “Removing the teeth of the top lama.” Additional qualitative results for drastic manipulation in a low-level manner caused by the edge primitive would be necessary.\n-\tI think the technical novelty of TPS-based augmentation in the paper is not significant in that the TPS transformation has been widely used in existing literature [2][3] for learning correspondence between two images.\n-\tAs the authors mentioned in the conclusion, training a network for a single image manipulation would be a critical bottleneck in practical use. In this respect, the training time on a single image should be reported. Additionally, detailed descriptions how to obtain the primitives (edge and segmentation) for the input image would be required.\n\n[1] Johnson et al., “Perceptual Losses for Real-Time Style Transfer and Super-Resolution.”, ECCV’16\n\n[2] Han et al., “VITON: An Image-based Virtual Try-on Network.”, CVPR’18\n\n[3] Lee et al., “Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence.”, CVPR’20\n\n-------------------------------------\nAfter rebuttal:\n\nThank you for the dedicated consideration of my comments, but there are a few remaining concerns that are not clear.\n\n1. The editing effects of edge maps are not distinct from those of segmentation maps. More specifically, except for the background in Fig.3 and the second face in App.D, most of the examples demonstrate the same kinds of manipulation as shown in the samples of the segmentation map manipulations. This includes moving, stretching, and erasing the objects. I think that the qualitative results of edge modification are not sufficient to prove its effectiveness, compared to those of segmentation maps.\n\n2. As shown in the image segmentation video the authors provide, a user needs to segment every single object which are selected for the manipulation. Moreover, segmenting small and fine objects requires further elaborate and laborious annotations from the user, resulting in a critical bottleneck for practical use.\n\nDue to these concerns, I would keep my previous rating of “6. Marginally above acceptance threshold.”\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews are a bit mixed. While all the reviewers feel that the paper proposed an interesting mechanism to train conditional generators from a single image and demonstrated good image editing results in the experiments, there are also common concerns about the practicality of the proposed method for interactive image editing. All the reviewers asked for the computation time, and some expressed the concerns about technical contributions. While these concerns were (somewhat) addressed in the rebuttal, the AC feels that it’s a hard sell to bet on the dramatic increase of computational capacity to make the computing time from an hour to realtime. Concerns about novelty also remained. Given the drawbacks, the final decision was to not accept. However, this work is promising and can be made stronger for publication in a later venue.
Decision: Reject