Generating Novel Scene Compositions from Single Images and Videos
Authors: Vadim Sushko
Dan Zhang
Juergen Gall
Anna Khoreva
Venue: ICLR 2022 Submitted (2022)
Keywords: GANs
Image Generation
Deep Learning
Image Synthesis
Generative Models

Abstract: Given a large dataset for training, GANs can achieve remarkable performance for the image synthesis task. However, training GANs in extremely low data regimes remains a challenge, as overfitting often occurs, leading to memorization or training divergence. In this work, we introduce SIV-GAN, an unconditional generative model that can generate new scene compositions from a single training image or a single video clip. We propose a two-branch discriminator architecture, with content and layout branches designed to judge internal content and scene layout realism separately from each other. This discriminator design enables synthesis of visually plausible, novel compositions of a scene, with varying content and layout, while preserving the context of the original sample. Compared to previous single image GANs, our model generates more diverse, higher quality images, while not being restricted to a single image setting. We show that SIV-GAN successfully deals with a new challenging task of learning from a single video, for which prior GAN models fail to achieve synthesis of both high quality and diversity.

Under review as a conference paper at ICLR 2022
GENERATING NOVEL SCENE COMPOSITIONS FROM SINGLE IMAGES AND VIDEOS
Anonymous authors Paper under double-blind review
ABSTRACT
Given large dataset for training, GANs can achieve remarkable performance for the image synthesis task. However, training GANs in extremely low data regimes remains a challenge, as overfitting often occurs, leading to memorization or train- ing divergence. In this work, we introduce SIV-GAN, an unconditional generative model that can generate new scene compositions from a single training image or single video clip. We propose two-branch discriminator architecture, with content and layout branches designed to judge internal content and scene layout realism separately from each other. This discriminator design enables synthesis of visually plausible, novel compositions of scene, with varying content and layout, while preserving the context of the original sample. Compared previous single- image GANs, our model generates more diverse, higher quality images, while not being restricted deals with new challenging task of learning from single video, for which prior GAN models fail to achieve synthesis of both high quality and diversity.
single image setting. We show that SIV-GAN successfully
Training video and samples generated from a single video
Training image and samples generated from single image
Images generated by SIV-GAN. Our model successfully operates extremely low data regimes, generating
Figure 1:
scene compositions with varying content and layout from single video (first two rows) or single image (last three rows). For example, from the single training video with a car on the road, SIV-GAN generates images without car with cars; and for the single surfing image, it can synthesize layouts with different position and configuration of waves varying the number of surfers the scene. SIV-GAN is able to maintain the quality and diversity of scene compositions while operating at different image resolutions, e.g., 192x 320 (fourth row) and 512x 896 (last row). (Original training samples are shown in grey or red frames.)

---

Under review as a conference paper at ICLR 2022
INTRODUCTION
The quality of synthetic images produced by generative adversarial networks (GANs) Goodfellow
al. 2014) has greatly improved in recent years (Zhang et al., 2019 Brock et al. 2019 Park al.
Karras et al.] 2019; Lin et al.) 2019 Schönfeld et al. 2020 K Karras et al. 2020b). These impressive results are in large part enabled by the availability of large, diverse datasets, typically consisting of tens of thousands of images. This dependency on the availability of training data limits the applicability of GANs in domains where collecting a large dataset is not feasible. In some real- world applications, collection of even a small dataset remains challenging due to specific constraints related to privacy, copyright status, subject type, geographical location, time, and dangerous or hazardous environments. It may happen that rare objects or events are present only in one image or video, and it is difficult to obtain a second one. For example, this includes images of exclusive artworks or videos of traffic accidents recorded in extreme conditions. Enabling GANs to learn in the extremely low data regimes, such as learning from the single data instance examples, thus has the potential improve their utilization in practice.
2019
Prior work (Shocher et al.] 2019; Shaham et al. 2019; Hinz et al. 2021) focused only on learning generative models from a single image (Fig. [ rows 3-5). In this work, we introduce a new task of learning to synthesize images from a single video (Fig. D rows 1-2), where the training data is collection of frames from a 2-10 second video clip. In practice, capturing such a short video takes almost little effort as collecting single image. However, compared to single image, video contains much more information about the scene and the objects of interest (e.g., different poses and locations of objects, various camera views, dynamic backgrounds). Learning from video can enable synthesis of higher quality, more diverse images, while still keeping very low requirements on data availability, and therefore can improve GAN usability for practical applications.
Training GANs in such extremely low data regimes is challenging the models are prone to overfit- ting. leading to replication of training data or poor synthesis quality due to training instability (Karras jet al. 2020a). For example, applying the few-shot image synthesis models, such FastGAN (Liu
2021). to learn from a single image or video leads to severe memorization problems (see Sec. let al.
4). To mitigate memorization, single-image GAN models (Shaham et al. 2019 Hinz et al., (021) proposed to learn an internal patch-based distribution of an image, employing a cascade of multi- scale patch-GANs (Isola et al. 2017) trained in multiple stages. Though these models overcome memorization, producing different versions of a training image. they cannot learn high-level seman- tic properties of the scene, e.g., to dissect objects from the background. They often suffer from incoherent shuffling of image patches, distorting objects and producing unrealistic layouts (see Fig. a Furthermore, they have more difficulty to generate realistic scenes from single video.
We aim to go beyond patch-based learning, seeking to generate novel, plausible compositions of objects in the scene, while preserving the original image context. We want to learn a model which is able to compose new layouts, re-arrange objects in the scene, remove duplicate instances, and change their shape and size. The new scene compositions should be visually plausible, with objects preserving their appearance and natural shape, and scene layouts looking realistic to the human eye. Reaching such synthesis quality from a single image or a single video is notably challenging, as the model needs to learn to distinguish objects and background using only one image or few training frames, without any direct object supervision (e.g., mask annotations).
To this end, we introduce SIV-GAN (Single Image and Video GAN), an unconditional, one-stage GAN model, capable of learning from the single data instances to generate images that are substan- tially different from the original training sample, while still preserving its context. This is achieved by two key ingredients: the novel discriminator design and the proposed diversity regularization for the generator. Our discriminator has two branches, separately judging image content and scene lay- out realism. The content branch evaluates objects' fidelity irrespective of their spatial arrangement, while the layout branch looks only at the global scene coherency. Disentangling the discrimina- tor's decision about content and layout helps to prevent overfitting and provides more informative signal to the generator. To achieve high diversity among generated samples, we further extend the regularization technique of (Yang et al., 2019) to unconditional image synthesis in single data instance regimes. The prior work of (Yang et al. 2019) encourages the generation of different im- ages depending on their input latent codes, thus the difference between images is proportional to the distance between their codes in the latent space. Assuming that in case of a single image single video all generated images should belong to one semantic domain (i.e. preserve the original training sample context), and thus should be more or less equally different from each other, we apply
2

---

the Under review as a conference paper at ICLR 2022
diversity regularization uniformly, independent of the latent space distance. Moreover, regularization in the feature space, inducing both high- and low-level diversity.
We demonstrate the effectiveness of our model for the single image setting as well as for the novel single video setting in Sec. 4 SIV-GAN is the first model that successfully learns in both of these
extremely low-data settings, improving over prior work (Shaham et al., [019 Hinz et al. 2021 Liu
et [ 2021) in both image quality and diversity. In contrast to FastGAN [Liu al. (021). our model does not suffer from memorization (see aad while compared to single-image GANs (Sha-) ham et al. 2019; Hinz et al. 2021), our model does not distort objects, preserving their appearance. In summary, our contributions are: 1) We propose a new task of learning generative models from a single video. 2) We present novel two-branch discriminator, encouraging the generation of new scene compositions with layouts and content substantially different from training samples. Our pro- posed diversity regularization ensures a high variability among generated samples in the challenging single data instance regimes. 3) With SIV-GAN, we achieve high quality and diversity when learning from both single images and videos, outperforming prior work on two different datasets.
2 RELATED WORK
Single Image Generative Models, Several works have investigated learning generative models from single image. [Uyanov al. (2018); Shocher et al. (2018) showed that when trained on a sin- gle image, a deep convolutional network can learn a useful representation that captures the internal statisties of that image. These learned representations can be employed to synthesize textures from sample texture image (Jetchev et al., 2016, Bergmann et al. 2017 Ulyanov et 2016; Zhou et al. 2018 Li & Wand 2016
Ulyanov et al. 2017). to "blindly" super-resolve (Shocher et al., 2018 Bell-Kligler et al. 2019 Lugmayr et al., 2020). or to impaint the image (Ulyanov et al. 2018)) Re- cently, single image GAN models (Shocher et al. 2019; (Shaham et al.] (211)) have been proposed, revealing the power of image priors learned from a single natural image for synthesis and manipu- lation tasks. InGAN (Shocher et al. (019) introduced a GAN model conditioned on a single natural image, which can remap the input to any size or shape while preserving its internal patch-based distribution. SinGAN (Shaham et al. 2019) employed an unconditional GAN to produce images of arbitrary size from noise, and used multi-stage training scheme learn the multi-scale patch dis- tribution of an image. ConSinGAN (Hinz et al., 202T) extended SinGAN by improving the rescaling for multi-stage training and training several stages concurrently, which enabled reducing the size of the model and making the training more efficient. Alternatively, instead of using adversarial training search.
Granot et al. (202I) proposed use patch-nearest-neighhee
Contrary to single image GANS (Shaham et al.] 2019] Hinz et al. (021), our model is trained in a single stage, and is designed to learn not only the internal patch-based distribution of an image, but also to capture high-level content, such as scene layouts appearance of objects. The latter enables more diverse, higher quality synthesis (see Table J. Our model is also more universal: it achieves high quality and diversity when learning from both single images and videos, and thus can be applied to a wider range of real-world scenarios.
Few-Shot Generative Models. Few-shot image generation seeks to generate more data of a given domain, with only few training samples available. Many existing methods resorted to using large image sets of seen classes to train conditional GAN, which is then used to generate new images for unseen class based on few examples (Hong et al. 2020a]] Antoniou et al. 2017 Hong et al.) 2020b). Others focused on improving the sample efficiency of GANs by few-shot adaptation. These methods usually start from GANs pre- are-trained large datasets, and adapt them
few samples in the target domain by fine-tuning the generator and discriminator weights (Wang et al., 2018 Mo et al. 2020), changing batch statistics (Noguchi & Harada 2019), restricting the space of trainable parameters (Robb et al. (020). transferring the knowledge via a miner network (Wang et al.) 2020), or enforcing cross-domain consistency (Ojha et al.] (021). Unlike our work, the above methods heavily rely on external knowledge from a pre-trained GAN. Thus, their performance highly de- pends on the semantic consistency between the target domain and the pre-trained model, and their incompatibility can lead to even worse results (Zhao et al. (220aa).
Most recently, FastGAN (Liu et al, 202T proposed to train an unconditional GAN for few-shot image synthesis from scratch. To avoid overfitting, they enabled fast learning of the generator through a skip-layer channel-wise excitation module, and used a self-super discriminator to continuously provide useful signals to the generator. However, FastGAN still suffers to successfully
3

---

Under review as a conference paper at ICLR 2022
Generator
Discriminator
Noise
Real
Low-level Features
Content Branch
Posling
Fake
Layout Branch
W
(tw))
Cayaa
W
Figure 2: SIV-GAN. The two-branch discriminator judges content separately from the scene layout
realism and thus enables the generator to produce images with varying content and global layouts.
learn from single image video. Although the single video setting has similar number of training frames to the shw- setting (~1100 samples), provides much less variability in training data due to high correlation of adjacent frames, leading FastGAN to memorization issues (see Fig. 5)
Another line of work focused on improving the stability of GANs when trained with limited data. To prevent the discriminator from overfitting, Karras et al. (2020a): Zhao al. (2020b a) explored differentiable data augmentations (DA) for both real and generated images. [seng et al. (2021) proposed regularize GANs under limited data with a LeCam consistency loss, showing it to be complementary to DA techniques. These works used limited, but still relatively large training sets C> 1K images) compared to the few-shot and single video regimes (& 100). As shown in Sec. [4. using DA alone is not enough to achieve good diversity when learning from single image video.
SIV-GAN
In this section, we present SIV-GAN, an unconditional GAN model that learns from a single image or single video to generate plausible compositions of given scene with varying content and layout. The key ingredients of SIV-GAN are a novel design a two-branch discriminator (Sec. 3.I
and diversity regularization introduced for synthesis in single data instance regimes (Sec. 3.2)
SinGAN, lowest resolution of 20px
3.1 CONTENT-LAYOUT DISCRIMINATOR
One challenge of training GANs in single data instance regimes is the problem of overfitting to original samples. In many cases the model can simply memorize the original training images and their augmented versions (if used during the training). To avoid this memorization ef- fect, Shaham et al. (2019) and et al. (2021) proposed learn internal patch-based dis- tribution of a single image by using a hier- archy of patch-GANs (Isola et al.) 2017 Li & Wand 2016) at different image scales. As the employed patch-GANS have small receptive fields and limited capacity. they are prevented from memorizing the full image. However, the downside of training each scale of the patch-GANs in separate stage is that any layout decisions made by the coarser scale generators cannot be corrected at later, finer generation stages. Thus, both the quality and diversity of generated images size. This parameter needs careful tuning for specific images at hand, as otherwise image layouts may exhibit lack of diversity loss of global coherency (see Fig. B) Moreover, this approach does not generalize to learning from multiple images, as in the single video case (see Table Fig.
Training image
SinGAN, lowest resolution of 35px
Figure 3: Limitation of the multi- stati-stage training of single image patch-GAN methods. As the finer generation stages cannot correct the layout deci- sion made by the coarser scale generators, with- out a careful tuning of the lowest resolution size the model produces images of very low diversity (first row) or lacking global coherency (last row).
highly dependent on the chosen lowest resolution
We therefore introduce an alternative solution to overcome the memorization effect but still to pro- duce high-quality images. We note that in order to produce realistic and diverse images, the gener- ator should learn the appearance of objects and combine them in the image in a globally-coherent way. To this end, we propose discriminator that judges the content distribution of a given image separately from its layout realism. To achieve the disentanglement, we design a two-branch dis- criminator architecture, with separate content and layout branches. Note that the branching of the discriminator happens after intermediate layers; this done in order to learn relevant representations for building the branches. As seen from Fig. 2 our discriminator consists of the low-level feature
4

---

Under review as a conference paper at ICLR 2022
extractor Diow-level. the content branch Deontent, and the layout branch Diogous. For given image I, the purpose of Diow-level is to learn low-level features and to produce an image representation F(z) Ptore-teves (z) for the branches. Next, Dcontent will judge the content of F(x), irrespective from its spatial layout, while on the other hand Diayout will inspect only the spatial information extracted from F(x). Inspired by the attention modules of Park et al. (2018) and Woo et al. (2018), we implement the content-layout disentanglement by squeezing channels or spatial dimensions of the intermediate features F(x). Note that afterwards the branches Doontent and Diagout receive only limited information about the image from F F(z), preventing them from overfitting to the whole image, and thus mitigating the negative effect of memorizing the original image.
Content branch. The content branch decision should be based upon the image content, i.e. the fidelity of objects composing the image. independent of their spatial location in the scene. Let the feature map F (x) have dimensions (hoight) x W (width) x C(channel). Note that the spatial dimensions H x W capture spatial information, while the channels C' encode the semantic repre- sentation. As we want the content branch ignore the spatial location of objects, we apply global average pooling to aggregate the spatial information H X W across the channels C. The resulting feature map F Feontent (z) has size 1 x 1 X C, which is then processed by several layers for further real/fake decision making. By removing the spatial information, D Deontent (1:) is induced to respond to content features encoded in different channels regardless of their spatial location (see App. App. [A.T).
Layout branch. The layout branch, in contrast, should assess the spatial location of objects in the scene, but not their specific appearance. Thus, the layout branch is designed to judge only the spatial information of F(z). filtering out the content details. Since the layout information is encoded only in spatial dimensions H x W, and not in channels C, we aggregate the channel information from F(x) via a (1 x 1) convolution with only one output channel, which forms feature map Flayona (1) with size H x W x1. 1. This channel aggregation weakens the content representation but does not affect the spatial information. The Flayout(z)) (z) features are further processed by several layers before a real/fake decision is made. As Drayout(2 (1)) is designed to be sensitive only to the spatial representation of the input image, it learns to judge the realism of scene layouts (see App.A.1
Feature augmentation. The proposed two-branch discriminator prevents the memorization of training samples, enabling the generation of images with content and layouts different from the original sample. To further improve the diversity of generated images, we propose augment the content F (ro)tene(r) and layout Floyous (z) features of real images. For the single image setting this done by mixing the features of two different augmentations of the original image, and for the sin- gle video setting by mixing the features of augmentations of two different video frames. For two real samples 21 and Ø2 we apply a mixing transformation F. (x1) = Tii ( (F= (21). ) F- (Z2)). We use two types of mixing: 1) For the layout branch, we sample rectangular crop of F Fingout (22) and paste it on to Flayout (11) at the same spatial location, similarly to CutMix (Yun et al., 2019). In contrast to CutMix, our approach augments features, not input images, and mixes only features of real images. 2) For the content branch, we sample a set of channels from Foontent (z2) and copy their values to the corresponding channels of F'conten (x1). As the channels encode semantic features of images, we expect the resulting augmented tensor to represent objects seen in two different images. We also found it useful to remove channels from Feontent (2), thus removing some object representations. For this, we sample a set of channels and drop out their values (Srivastava et al. [0144] Zhengsu et al.) 2018). With the above augmentations, Deontent and Diayout see significantly more variance in both the content and layout representations of real images, which prevents overfitting and improves the augmentuu diversity of generated samples. The feature augmentation (FA) effect is shown in Table 3
Adversarial loss. To evaluate images at different scales, we design our discriminator to make bi- nary true/fake decision at each intermediate resolution. For each discriminator part D.: Diow-tevels D Doontent, and Dtayout, the loss is computed by aggregating the contributions across all layers con- stituting the corresponding discriminator part:
1 N.
LD. No 1=1
M DD!
(1)
where D., is the 1-th ResNet block of D., N. is the number of ResNet blocks used in D., and the loss LDz is the binary cross-entropy: LD! EEol
DJ aims to distinguish between real and generated G(z) images based on their corresponding features at block 1, which captures either their low-level details, content, or layout at a certain resolution. The overall adversarial loss for SIV-GAN is then computed by taking the decisions from the content
Eollo((
5

---

Under review as a conference paper at ICLR 2022
branch D Dontentt the layout branch Diagout, and the low-level features of Diow-level:
LG Doontenn + Dingoue + 2 2C Diowstavet
(2)
where D aims to distinguish between real and generated images based on their low-level, content, and layout realism. As the two discriminator branches operate on high-level image features, contrary to only one Diow-tevet operating on low-level features, we double the weighting for Diooweeust This done in order to properly balance the contributions of different feature scales and encourage the generations of images with good low-level details, plausible content, and coherent scene layouts.
3.2 DIVERSITY REGULARIZATION
To improve the variability among the generated images, propose to add diversity regularization (DR) loss term LDR to the SIV-GAN objective. Prior work (Yang et al. 2019) Zhhao et al.) [02T]]
Choi et al. 2020) also proposed to use diversity regularization for GAN training, but mainly to avoid mode collapse, and assuming the availability of a large training set. The regularization of Yang al. (2019) aimed encourage the generator to produce different outputs depending on the input latent code, in such a way that generated samples with closer latent codes should look more similar to each other, and vice versa. In contrast, our diversity regularization is tuned for synthesis in single data instance regimes. Assuming that in case of a single image video we are operating in one semantic domain, the generator should produce images that are in- domain but more or less equally different from each other, and substantially different from the original training sample. Thus, in such regimes the difference of generated images should not be dependent the distance between their latent codes, so propose encourage the generator to produce perceptually different image samples independent of their distance in the latent space. Mathematically, the new diversity regularization expressed as:
single
LDR(G) = = 22 1 Σ
where || denotes the L1 norm, (z) indicates a feature extracted after (-th resolution block of the generator G given input z, and æ 22 are randomly sampled latent codes in the batch, i.c. 21 22 ~ N 1). By regularizing the generator to maximize Eq. 3 we force it produce diverse outputs for different latent codes z. Note that, in contrast to Yang et al. (019); ;; Zhao et al. (021): Choi et al. 2220)). we compute the distance between samples in the feature space of the generator. Computing the distance in the feature space results in more meaningful diversity among the generated images,
as different layers of the generator capture different image semanties, inducing both high-level and
low-level diversity, Computing the distance in the image space i.c. GDR(G)) = [[G(21) - G(2))
as in Choi al. (2020), leads to reduced image diversity in our experiments (see Table
The overall SIV-GAN objective can be written as:
max Ludo + ACDn(G),
(4)
where \ controls the strength of the diversity regularization and Ladv is the adversarial loss in Eq. 2 The proposed diversity regularization is shown to be highly-effective for SIV-GAN, while prior
regularizations (Yang et al. 2019: Zhao al. 2021
Choi al. 2020) underperform (see Table 4
3.3 IMPLEMENTATION AND TRAINING
The SIV-GAN generator employs ResNet blocks, similar to BigGAN (Brock et al. 2019). However, do not use BatchNorm or self-attention. As in MSG-GAN (Karnewar & Wang 2020). gener- ate images at intermediate ResNet blocks of G. passing them to Dtow-tevel to facilitate the gradient flow from the discriminator. The latent vector of length 64, is sampled from NO, 1). For diversity regularization we use the tanh activation on the features from the final convolutions of the G blocks.
The SIV-GAN discriminator also uses ResNet blocks. We set Niow-level = 3, Ntayout Neontent = 4, thus using 3 ResNet blocks before branching and 4 ResNet blocks for the con- tent and layout branches. To enable multi-scale gradients, we incorporate images at different scales using the Wlin_ost strategy from Karnewar & Wang (2020). The proposed feature augmentation (FA) is applied with probability 0.4 at every discriminator forward pass. We also use differentiable image augmentation (DA) (Karras et al.) [020a]] Zhao al. 2020b a). applying translation, cropping, rota- tion, and horizontal tlipping for real and take images with a probability of 0.7 at each forward pass. As in (Karras al. 2020a), observe signs of leaking augmentations in the generated samples.

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison with other methods in the Single Image setting on Places (Zhou et al.
2017a
and DAVIS- YFCC110M (Perazzi al.
2016
Thomee et al. 2016) datasets.
Places
DAVIS YFCCI00M
SIFID
Pixel Diversity
Dist.
SIFID
Pixel Diversity
Dist.
Method
LPIPSt
LPIPST
best
mean J
train
best
mean
train
SinGAN
0.09
0.15
0.22
0.52
0.24
0.10
0.13
0.26
0.54
0.30
ConSinGAN
0.06
0.08
0.24
0.50
0.25
0.08
0.09
0.29
0.59
0.31
FastGAN
0.11
0.14
0.15
0.48
0.08
0.10
0.13
0.18
0.49
0.11
SIV-GAN
0.05
0.06
0.28
0.57
0.31
0.00
0.08
0.33
0.66
0.37
SinGAN
ConSinGAN
SIV-GAN
Training frame
SinGAN
FastGAN
SIV-GAN
Training frame
Visual comparison in the Single Image setting. Single-image GANs (Shaham et al. 2019 Hinz et al., 2021) incoherently shuffle patches (e.g. sky textures below horizon, perturbed fish contours), and few-shot FastGAN (Liu et al. 2021) reproduces the original image its flipped version. In contrast, SIV-GAN achieves high diversity, maintaining content and layout realism.
Figure 4:
In contrast to previous single image GANs (Shaham et al. 2019 Hinz et al. 2021), which employ multi-stage training scheme, SIV-GAN is trained end-to-end in one stage, with the losses from Eg. with A = 0.15 for LDR (see the ablation on À in App I.I.. We use spectral normalization (Miyato
ctt al. 2018 for both G and D, and do not use a reconstruction loss as in (Shaham et al. 2019 Hinz any other stabilization techniques. SIV-GAN is trained using the ADAM optimizer with [31, 32) = (0.5, 0.999), a learning rate of 0.0002 for both G and D, and a batch size of 5 (using
al. 2021
different augmentations of single image video frames). SIV-GAN can operate at different image resolutions by simply changing the input noise shape (see App. E.2 successfully producing images of high quality and diversity at resolution 512x 896 (see Fig.
and Fig. for examples).
EXPERIMENTS
We evaluate SIV-GAN the Single Image and Single Video settings, using the model configuration image and 300k iterations video for training.
described in Sec. 3.3 We use 100k iterations
Datasets. Following Shaham et al. (219)). we evaluate the Single Image setting on 50 images from Places (Zhou al. 2017a). ). In addition to their protocol, we select videos from the DAVIS (Per-) azzi 2016) and YFCC100M (Thomee et al. (016)) datasets. In the Single Video setting, use all the frames as training images, while for the Single Image setup we use only one middle frame. The chosen videos last for 2-10 seconds and consist of 60-100 frames.
Metrics. To assess the quality of generated images, we measure the single FID (SIFID) (Shaham al. 2019). Its original formulation uses V3 features before the first pooling layer HXW
resolution, which capture low-level image details. To assess high-level properties, such as
scene layouts, also propose to compute SIFID at HxW resolution. To evaluate the diversity of samples, we adopt the pixel diversity metric from (Shaham et al. 2019). To measure perceptual diversity, we also report the average LPIPS (Dosovitskiy & Brox 2016) across pairs of generated images. To verify that the models do not simply reproduce the training set, we report average LPIPS to the nearest image in the training set, augmented in the same way during training (Dist. to train).

---

Under review as a conference paper at ICLR 2022
Video frames
SinGAN
SIV-GAN FastGAN
Figure 5: Visual comparison in the Single Video setting. While other models fall into reproducing the training frames or fail to correctly generate objects, SIV-GAN produces high-quality images substantially different from the original training frames.
We note that SIFID tends to penalize diversity, favouring overfitting (Robb et al. 2020). Thus, to account for this quality-diversity trade-off, a fair analysis should assess both diversity and quality.
Comparison models. We compare our model with single image methods. SinGAN (Shaham al. 2019) and ConSinGAN (Hinz et al. 2021), and with a recent model for few-shot image synthesis, FastGAN (Liu et al. 2021). We use the original implementation codes provided by the authors. While training single image GANs from (Shaham et al. 2019] Hinz et al.) (02T) on single video,
we applied the reconstruction loss on all frames, as we found this helpful in stabilizing the training.
4.1 MAIN RESULTS
Tables [ 2 present quantitative comparison be- tween the models in the Single Image and Single Video settings, while the respective visual results are shown Fig. and [ As seen from the ta- bles, SIV-GAN notably outperforms other models in both studied settings. Despite a potential trade-off between quality and diversity, our model achieves better performance in both, reaching lower SIFID values and higher diversity scores. Importantly, only SIV-GAN successfully learns from both single images and videos, generating globally-coherent im- ages of high diversity. Next, analyse results in these settings separately.
Table 2: Comparison in the Single Video set- ting DAVIS- YFCCIOOFC
SIFID
Dist. to train
Method
HxW
HxW
LPIPS
SinGAN
2.47
96.35
0.32
0.51
ConSinGAN
2.74
74.50
0.34
0.53
FasIGAN
0.79
9.24
0.43
0.13
SIV-GAN
0.55
5.14
0.43
0.34
Single Image. As seen from Fig. D and Fig. 4 in the Single Image setting, SIV-GAN produces diverse, high-quality images. For example, our model can change the number and placement of balloons in the sky, or edit the contour and position of rocks in the desert. Note that such changes retain the appearance of objects, preserve original context, and maintain scene layout realism. In contrast, the prior single-image GAN models, SinGAN and ConSinGAN, disturb the appearance of objects (e.g. by washing away fish contours) and disrespect layouts (c.g. sky textures appear below the horizon), while having modest diversity. This is reflected in their higher SIFID and lower diversity scores in Table I The few-shot FastGAN model suffers from memorization, reproducing the training image its flipped version. In Table this is reflected in lowest diversity and Dist. to train (in red) metrics on both datasets. Despite having the lowest diversity, we observe that FastGAN does not reach low SIFID due to leaking augmentations (horizontal flipping).
Single Video. In the Single Video setting, there is much more information to learn from, and gen- erative models can consequently synthesize more interesting combinations of objects and scenes. Fig. and Fig. 5 show the images generated by SIV-GAN in this setting. Our model generates high- quality images that are substantially different from the training frames, adding/removing objects and changing scene geometry. For example, having seen car following road (Fig. D.. SIV-GAN generates the scene without a car or with two cars. In Fig. 5 our model varies the length of a bus and placement of trees, removes horse from the scene and changes the jumping obstacle config-
8

---

Under review as a conference paper at ICLR 2022
Table 3: Ablation study in the Single Image and Video settings on -FVII--YFCIIOMM, Indicators
of collapsed diversity (low LPIPS, Pixel Diversity) or poor quality (high SIFID) are marked in red.
Single Image
Single Video
SIFID
Pixel t
Dist.
SIFID
Dist. to train
Method
HxW
LPIPS
diversity
to train
HxW
Hxw
LPIPS f
14
18
Full model
0.08
16.30
0.33
0.66
0.37
0.55
5.14
0.43
0.34
No Layout hr.
0.14
20.29
0.35
0.67
0.40
0.71
11.70
0.42
0.38
No Content br.
0.08
23.25
0.34
0.64
0.36
0.73
10.43
0.41
0.33
No branches
0.03
7.73
0.13
0.43
0.12
0.42
3.73
0.37
0.18
No DR
0.05
11.99
0.04
0.33
0.06
0.40
9.81
0.30
0.32
No FA
0,08
14,81
0.27
0,58
0.33
0.51
4.85
0,41
0.32
No D
0.08
15.92
0.27
0.56
0.29
0.58
5.32
0.40
0.31
uration. In contrast, SinGAN, tuned to learn from single image, does not generalize to the Single Video setting, distorting objects and producing unrealistic scene layouts (low diversity and an tremely high SIFID in Table 2) FastGAN, on the other hand, generates images with reasonable fidelity, but fails to produce samples with non-trivial layout changes, having very low Dist. to train score (0.13 in Table 2. Additional qualitative results can be found in App. D.1
4.2 ABLATIONS
In Table g we demonstrate the importance of the proposed modifications. In each line we remove only one model component, starting from our full model. Firstly, we ablate our discriminator architec- ture, testing it without any branches (No branches), corresponding to standard GAN discriminator, and without the layout branch or the content branch. The model without branches trained together with our
Table 4: Comparison of diversity regulariza- tion techniques in the Single Image setting on RAVIS-YFCCION
Pixel Diversity
Regularization
SIFID + LPIPS 1
None zCR
0.05
0.04
0.33
0.05
0.06
0.37
DS DR (im. space)
0.06
0.14
0.45
proposed DR and FA, as well as differentiable aug- mentations (DA) as in (Karras et al.) 2020at Zhao
0.07
0.21
0.52
DR
0.08
0.33
0.66
et al. 2020a). However, as seen from Table 3 it
memorizes the training images and reproduces them with poor diversity. Using only one of the branches shows good diversity, but the model fails to generate globally-coherent images, having high HxW SIFID. The qualitative results for these ablation models, as well as additional analysis of branches learning content and layout, are presented in App. A. Next, we observe the effect of the proposed diversity regularization (DR), feature augmentation (FA), and low-level loss Dio Without DR, the model does not achieve multi-modality, scoring low diversity metrics. The absence of FA notably decreases diversity, which is reflected in the diversity scores dropping by 0.02-0.08 points. Removing Lirwwstoust also reduces diversity; we discuss this effect in App. [A.4 Overall, we observe that the success of our model in both settings is enabled by both the two-branch discriminator and the diversity regularization. In App.Cwe we also show the benefit of using SIV-GAN discriminator for few-shot image synthesis, outperforming the FastGAN model.
16
In Table 4 we compare our DR to the latent consistency regularization (ZCR) Zhao et al. (202)) and diversity-sensitivee loss (DS) Yang al. 2019). Our DR noticcably improves over zCR and DS on all diversity metrics. We find it beneficial to use DR in the feature space, which leads to more
variation in the generated samples. Interestingly, Table 4 illustrates quality-diversity trade-off, where improvements in diversity lead to deterioration in SIFID. See more ablations in App. B
CONCLUSION
We propose SIV-GAN, a new unconditional generative model which successfully learns from a sin- gle image or a single video. such extremely low-data regimes, our model prevents memorization and generates diverse images that are significantly different from the training set. Inherently, the synthesis of our model is constrained by the appearance of objects present in the original sample. Nevertheless, SIV-GAN can synthesize novel scene compositions by blending objects in different combinations, changing their shape or position, while preserving the original context and plausi- bility of the scene. Such compositionality is enabled by the model's ability to distinguish objects and backgrounds, astonishingly, while learning from a single image or video. We believe SIV-GAN provides a useful tool for data augmentation in domains where data collection remains challenging.
9

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
All the authors of this work adhered to the ICLR Code of Ethics at the time of preparing the submis- sion. We have made our best effort to uphold high standards of scientific excellence, avoid harm, discrimination, and dishonesty. All the experiments presented in the paper conducted on pub- liely available datasets with a granted right for research-purposed usage. Our organization is carbon neutral so that all its activities including research actívities do longer leave carbon footprint. This also includes our GPU clusters on which the experiments have been performed.
The method presented in this work allows high-quality image synthesis in extremely low data regimes. Our model allows generating novel scene compositions requiring only single image or video for training. This ability can be potentially utilized in different applications, ranging from Al-based content editing to data augmentation of limited domains. For the content creation appli- cations, similarly to other data-driven editing tools and graphical engines, a potential concern of misuse is the arising possibility for data manipulation. Photorealistic fake images synthesized by GANs may be presented as real photography in the media coverage, which may lead to negative societal outcomes. For the data augmentation, our model trained on rare samples generate more examples of underrepresented data, thus upsampling its proportion in the original data collection. This step will help to weaken the bias of datasets, providing more fairness in data coverage.
be used to
REPRODUCIBILITY STATEMENT
The authors have made their best effort to ensure the reproducibility of the methods presented in this work. A complete description of algorithmic proposals is presented in Sec. B and our experi- mental and evaluation settings are discussed in Sec. 4 An extended description of all architectural solutions, training details, including detailed scheme of our networks is included in App. E To facilitate the usage of our models and the reproduction of our results, we will release publicly our training code and trained model weights, providing complete list of images and videos used for experiments.
REFERENCES
A. Antoniou, A. Storkey, and H. Edwards. Data augmentation generative adversarial networks. In International Conference on Learning Representations (ICLR), 2017.
Sefi Kligle, Assaf Shocher, and M. Irani. Blind super-resolution kernel estimation using an internal-gan. In Advances in Neural Information Processing Systems (NeurlPS), 2019.
Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. Learning texture manifolds with the periodic spatial gan. In International Conference on Machine Learning (ICML), 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems (NeurIPS). 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2014.
Niv Granot, Assaf Shocher, Ben Feinstein, Shai Bagon, and Michal Irani. Drop the gan: In defense of patches nearest neighbors as single image generative models. arXiv:2103. 15545, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurlPS). 2017.
10

---

Under review as a conference paper at ICLR 2022
Tobias Hinz, Matthew Fisher, Oliver Wang, and Stefan Wermter. Improved techniques for training single-image gans. In Winter Conference on Applications of Computer Vision (WACV), 2021.
Y. Hong, Li Niu, J. Zhang, Jing Liang, and L. Zhang. Deltagan: Towards diverse few-shot image generation with sample-specific delta. Stt1:2009.087553 2020a.
Y. Hong, Li Niu, J. Zhang, Weijie Zhao, Chen Fu, and Zhang. F2gan: desiin-anddfilling gan for few-shot image generation. In ACM International Conference on Multimedia, 2020b.
Phillip Isola, Jan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf. Texture synthesis with spatial generative adversarial networks. In NeurIPS Workshop on Adversarial Training. 2016.
Animesh Karnewar and Oliver Wang. Msg-gan: multi-scale gradient gan for stable image synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR),
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
Tero Karras, Miika Aittala, Janne Hellsten, S. Laine, J. Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems (NeurIPS), 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Ana- lyzing and improving the image quality of StyleGAN. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Chuan Li and M. Wand. Precomputed real-time texture synthesis with markovian generative adver- sarial networks. In European Conference on Computer Vision (ECCV), 2016.
Chieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann-Tzong Chen. Coco-gan: Generation by parts via conditional coordinating. In International Conference on Computer Vision (ICCV), 2019.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations (ICLR), 2021.
Andreas Lugmayr, Martin Danelljan, Lue Van Gool, and Radu Timofte. Srflow: Learning the super- resolution space with normalizing flow. In European Conference Computer Vision (ECCV), 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze discriminator: A simple baseline for fine-tuning gans. In CVPR Al for Content Creation Workshop, 2020.
Atsuhiro Noguchi and T. Harada. Image generation from small datasets via batch statistics adapta- tion. In International Conference on Computer Vision (ICCV), 2019.
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Jongchan Park, Sanghyun Woo, Joon- Young Lee, and In So Kweon. Bam: Bottleneck attention module. In British Machine Vision Conference (BMVC), 2018
11

---

Under review as a conference paper at ICLR 2022
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Federico Perazzi, Jordi Pont- Brian Mc McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine- Horiine benchmark dataset and evaluation methodology for video object segmenta- tion. In Conference Computer Vision and Pattern Recognition (CVPR), 2016.
E. Robb, Wensheng Chu, A. Kumar, and J. Huang. Few-shot adaptation of generative adversarial networks. arXiv:2010. 1943, 2020.
Edgar Schönfeld, Bernt Schicle, and Anna Khoreva. A u-net based discriminator for generative adversarial networks. In Conference Computer Vision and Puttern Recognition (CVPR), 2020.
Tamar Rott Shaham, Tali Dekel, and T. Michaeli. Singan: Learning generative model from single natural image. In International Conference on Computer Vision (ICCV), 2019. 2019.
Assaf Shocher, N. Cohen, and M. Irani. Zero-shot super-resolution using deep internal learning. In Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Assaf Shocher, S. Bagon, Phillip Isola, and M. Irani. Ingan: Capturing and retargeting the dna of natural image. International Conference on Computer Vision (ICCV), 2019.
Zhangzhang and Song-Chun Zhu. Learning hybrid image templates (hit) by information projec- tion. IEEE Transactions pattern analysis and machine intelligence, 2011.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 2014.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m The new data multimedia research. Communications of the ACM, 2016.
Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming- Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In Conference on Computer Vision and Pattern Recog- nition (CVPR), 2021.
D. Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempitsky. Texture networks: Feed-forward synthesis of textures and stylized images. In International Conference on Machine Learning (ICML), 2016.
D. Ulyanov, A. Vedaldi, and Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In Conference Computer Vision and Pattern Recognition (CVPR), 2017.
D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Yaxing Wang, Chenshen Wu, L. Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and B. Ra- ducanu. Transferring gans: generating images from limited data. In European Conference on Computer Vision (ECCV), 2018.
Yaxing Wang, Abel Lanzalez-Garcia David Berga, L. Herranz, F. Khan, and Joost de Wei- jer. Minegan: Effective knowledge transfer from gans to target domains with few images. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In European Conference on Computer Vision (ECCV), 2018.
Dingdong Yang, Scunghoon Hong, Y. Jang, T. Zhao, and H. Lec. tional generative adversarial networks. In International Conference Learning Representations (ICLR), 2019.
condi-
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv: 00.5550663366 2015.
12

---

Under review as a conference paper at ICLR 2022
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Interna- tional Conference on Computer Vision (ICCV), 2019.
Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine learning (ICML), 2019.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Conference on Neural Information Processing Systems (NeurIPS), 2020a.
Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for gan training. arXiv:2006.02595, 2020b.
Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Im- proved consistency regularization for gans. In Conference on Artificial Intelligecce (AAAT), 2021.
Tian Zhengsu, Chen Jianwei, and Niu Qi. Dropfilter: Dropout for convolutions. arXiv: 18100999,, 2018.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017a.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017b.
Yang Zhou, Zhen Zhu, X. Bai, Dani Lischinski, D. Cohen-Or, and Hui Huang. Non-stationary texture synthesis by adversarial expansion. In ACM Transactions Graphics (TOG), 2018.
13

---

Under review as a conference paper at ICLR 2022
APPENDIX
This supplementary material is structured as follows:
Additional analysis of the SIV-GAN discriminator
A.T
Effect of the two branches in the discriminator
A.2
Ablation on the number of discriminator blocks used before branching
A.3
Effect of the number of channels the layout branch
A.4
Effect of the low-level loss
B
Additional analysis of diversity regularization
B.
Ablation on the strength of the proposed diversity regularization
B.2
Effect of the proposed diversity regularization other GAN models
C
Results on few-shot image synthesis
Standard few-shot image synthesis benchmarks
Extremely low few-shot setting
Non-object-centric few-shot image synthesis
Additional
qualitative and quantitative results
D.1
Qualitative results the Single Image and the Single Video settings
D.2
High resolution image synthesis results
D.3
SIFID using different Inception V3 layers
D.4
Comparison of synthesis diversity in the Single Image and Single Video settings
D.5
Qualitative effect of the proposed diversity regularization and feature augmentation
B
Detailed description of the architecture and training details
Analysis of strategies generate images of rectangular shapes
Architecture and training details
A ADDITIONAL ANALYSIS OF THE SIV-GAN DISCRIMINATOR
A.1 EFFECT OF THE TWO BRANCHES THE DISCRIMINATOR
In Fig. A AandB we showcase the visual effect of using content and layout branches in the SIV-GAN discriminator. Following the ablation Table 3 in the main paper, we separately show qualitative re- sults for our full discriminator model with content and layout branches, the discriminator without the layout branch, without the content branch, and for a model without both branches, corresponding standard GAN discriminator. The visual results illustrate the concepts of "layout" and "content" learnt by the respective branches. Applying the content branch, without the layout branch, leads to the generation of different objects in various combinations, but the model often fails to reproduce correct positioning of objects or globally-coheren layouts. For example, there might be a horizon discontinuity, or air balloons may follow unrealistically structured positions in a grid. In contrast, model, trained with the layout branch, but without the content branch, generates images with more realistic layouts, but does not preserve the content distribution of the training image, removing ob- jects, distorting their appearance perturbing their shapes. Behaviour of such models corresponds to high diversity scores and poor low-level SIFID in Table Employing none of the branches, which corresponds to using standard GAN discriminator, leads memorization of the training image, so the model just reproduces the training data (low distance to training set in Table 3 Finally, our full model with two-branch discriminator generates plausible diverse images, varying both the image content and the global layout of the scene (0.33 LPIPS and SIFID at scale HxW of 16.3 in the Single Image setting, see Table 3.
16
To illustrate further our intuition on the discriminator's branches learning content and layout, analyse the feature distances between real images in the content and layout embeddings of the trained SIV-GAN discriminator. For this, take the discriminators trained on single "bus" and "parkour" videos from the DAVIS dataset (Perazzi et al., 2016). The video sequences have notably different variability in layouts: while there is significant camera direction movement for the "parkour"
14

---

Under review as a conference paper at ICLR 2022
No branches
No layout branch No content branch
Both branches
Training image
13
OF
No branches
No layout branch
No content branch
Both branches
Training image
No branches
No layout branch
No content branch
Both branches
Training image
Figure Visual effects of using content and layout branches in the discriminator in the Single Image setting. The model with standard GAN discriminator (No branches) memorizes the training image. Model without the layout branch fails to produce images with realistic layouts or positioning of objects. Absence of the content branch does not preserve well object appearances. Finally, the model with both branches generates diverse images with realistic content and layouts. See Table for qualitative comparison and App. A. for the discussion.
15

---

Under review as a conference paper at ICLR 2022
Video frumes
No branches
No layout
Both No brunches content
Figure B: Visual effects of using content and layout branches the discriminator in the Single Video setting. The model with standard GAN discriminator (No branches) memorizes the training video frames. The model without the layout branch is prone to cutting objects' contours, generating globally incoherent layouts. The absence of the content branch leads to failures in appearances of objects, such as trees. Finally, the model with both branches generates diverse images maintaining realistic content and layouts. See Table 3 for qualitative comparison and App. A. for the discussion.
Feature space of D trained on Bus video Content distance to parkour midile frame
First frame (-40)
space
First frame (-00)
Content distangg ball misd feomuu
distance the middle
Layour distance to "arrrour middle frame
Layout
Middle Frame (01
Middle frame (0)
Content
Layout Content
Last frame (40)
Last frame (40)
-10
Temporal distance to the middle Trame
Tomporal distance the middle frame
Figure C: Feature distances from video frames to middle frame. As nearby frames have very
similar content and layouts, the lowest distances are between adjacent frames. The layout embedding distances (solid red) between bus frames lower than for the parkour video, being in line with the visual layout variance of these sequences. As frames the short videos depict similar content (e.g- same objects), the content embedding distances between frames of the same sequence (solid blue) are significantly lower than distance between middle frames of different videos (dashed blue).
video, the "bus" video captures the vehicle always from the same angle. This leads to a much smaller perceptual differences in layouts for the bus video compared the parkour video.
In Fig. Ce show the feature distances in the embeddings of SIV-GAN content and layout branches, measured between the middle frame and other frames of the same video sequence. We observe that the illustrated distances correlate well with our intuition of D branches learning content and layout. Firstly, as nearby frames have very similar content and layouts, the lowest distances both in the content and layout embeddings are between adjacent frames, while increasing the temporal distance from the middle frames leads to higher content and layout feature distances. Secondly, the em- bedding differences are consistent with perceptual judgement: the layout embedding distances (solid red) between "bus" frames are notably lower than for the "parkour" video. Finally, as video frames inside the short videos depict similar content (e.g. same objects), the content embedding distances between frames of the same sequence (solid blue) are significantly lower than to middle frames of two different videos (dashed blue).
16

---

Under review as a conference paper at ICLR 2022
Training frame
N
Divndeve! N.
=2 N Denuli
3 N
Dionddeve
4 N
Diomiteunt
Figure Effect of the number of discriminator blocks used before branching. Using too few blocks (1-2) leads to reduced synthesis quality, as Dtow-level is unable to extract the features neces- sary to build the content and layout representations. Increasing the number of blocks (3-4) results in improved quality while maintaining good diversity. Using too many blocks (5-6) leads to the memorization effect due to the occurring overfitting, when the model reproduces the training image with diversity.
Table A: Ablation on the number of blocks Diowitene used before content-layour branching in the discriminator on the DAVIS- -FCICC000 dataset.
Single Image
Single Video SIFID
N Diowsteet
SIFID || LPIPS t
LPIPS 1
0.59
0.42
2.75
0.46
0.13
0.40
1.12
0.45
3
0.08
0.33
0.55
0.43
0.06
0.24
0.36
0.38
5
0.03
0.15
0.40
0.37
0.03
0.13
0.35
0.36
A.2 ABLATION ON THE NUMBER OF BLOCKS USED BEFORE BRANCHING
The proposed SIV-GAN discriminator has two branches, preceded by low-level feature extractor. As discussed in Sec. 3.3 the discriminator consists of 7 ResNet blocks, using ResNet blocks before the branching for Dion-Lenees and 4 ResNet blocks for the both branches Deontent and Dlayout- In Table[A]and Fig. Diwe analyse the effect of applying branching at an earlier or a later discrimination stage, keeping the overall depth of the networks equal to ResNet block. The results indicate that the branching should be applied neither too early nor too late. Using too few ResNet blocks (1-2) before the branching leads a reduced capacity of the low-level feature extractor Diow-Levels this network becomes unable to provide the branches with descriptive content and layout representations of an image. As seen from Fig. D such model learns the color distribution of an image, but cannot produce globally-coheren scene and generate textures of good quality. In Table A this effect is indicated by very high SIFID. On the other hand, using too many blocks before the branching (5-6) leads to increased capacity of the feature extractor Diow-tevel- so the network can remember the whole image. In this case the model suffers from a memorization effect, reproducing the training image (see Fig. DD. and scoring low at diversity metries (low LPIPS in Table Finally, found that using 3 ResNet blocks before the branching leads to an optimal quality-diversity trade-off in both the Single Image and Single Video settings, allowing the generation of high diversity, while preserving the context of original samples.
A.3 EFFECT OF THE NUMBER OF CHANNELS IN THE LAYOUT BRANCH
In this section we study the effect of varying the number channels the layout branch feature representation. For all experiments in the main paper, the layout branch representation is obtained from an intermediate representation by a convolutional layer with one output channel. Table B shows the effect of using the layout representation with 3, and 10 channels. An increased number of channels in the layout branch results in a more powerful network that can easier memorize the layout representations extracted from real images. Thus, increasing the number of channels results in reduced diversity of samples, as indicated by the smaller LPIPS scores in Table
A.4 EFFECT OF THE LOW-LEVEL LOSS
In this section we provide an ablation on the low-level discriminator loss Di According to Eq. [ and 2 the loss of the SIV-GAN diseriminator is computed after each ResNet block before branching, which means that the attention of the discriminator shared across different layers, corresponding to image realism different low-level scales. Without this loss, the discriminator
17

---

Under review as a conference paper at ICLR 2022
Table B: Ablation the number of channels used for the layout branch feature representation the DAVIS-YFCCIOOM dataset.
Playout channels
Single Image
Single Video SIFID
SIFID 4 LPIPS
LPIPS
0.08
0.33
0.55
0.43
0.07
0.28
0.54
0.41
5
0.06
0.25
0.50
0.40
10
0.06
0.24
0.49
0.40
Table C: Ablation on the low-level loss D on
dataset.
Low-level loss
Single Image
Single Video
SIFID . LPIPS t
SIFID
LPIPS
0.08
0.27
0.58 0.55
0.40
0.08
0.33
0.43
judges images only after branching, paying attention mostly to the high-level image realism. The low-level loss changes the discriminator's task and shifts its attention towards earlier layers, thus increasing its attention to low-level details, e.g., such textures. As seen from Table this property helps to reduce overfitting and allows synthesis of higher diversity, both in the Single Image and Single Video settings.
B ADDITIONAL ANALYSIS OF DIVERSITY REGULARIZZTION
B. ABLATION ON THE STRENGTH OF THE PROPOSED DIVERSITY REGULARIZATION
The proposed diversity regularization (DR) is an essential component for SIV-GAN to achieve high diversity among generated samples. For all our experiments in the main paper we used DR with A = 0.15. In Table D we show the effect of setting different À for DR in the Single Image setting, changing the strength of the diversity regularization. We note a general quality-diversity trade-off which is present for the Single Image setting. While diversity metrics favour diverse multi-modal outputs, the image quality metrics, such as SIFID, are computed based on similarity to the training frame, so they penalize variations from the original image (Robb al.) 2020).. Table D illustrates that setting the multiplier too high (A = 0.50) leads to good diversity. but harms image quality, while setting small values (A [0.00, 0.05)) is beneficial for quality, but deteriorates diversity. We observed that using À = 0,15 leads to good trade-off, resulting in a high diversity among generated samples, while corrupting the quality of textures and the global layout coherency, so we picked this value for the final version of the model.
B.2 EFFECT OF THE PROPOSED DIVERSITY REGULARIZATION ON OTHER GAN MODELS
To examine the effect of the proposed DR on other models, we trained SinGAN, ConSinGAN and FastGAN with the DR added to the GAN objective. Following design, this loss computed the feature space, as Eq. 3 When training the multi-stage single-image GANs from Shaham
al. (2019); Hinz al. (2021), 1b, applied the DR during training of each stage. To select X, tested the models with the values from Table D (0.05, 0.15, 0.50), and chose the highest coefficient
that improved diversity but did not lead to very high SIFID, indicating reduced low-level quality of generated images.
Table D: Effect the diversity regularization (DR) strength the Single Image setting on DAVIS- YFCC100M.
DR
Pixel 1 Diversity
Dist
SIFID
LPIPS 1 MS-SSIM 4
train
0.00
0.05
0.04
0.95
0.33
0.06
0.05
0.07
0.19
0.77
0.50
0.23
0.15
0.08
0.33
0.63
0.66
0.37
0.50
0.13
0.39
0.55
0.69
0.46
18

---

Under review as a conference paper at ICLR 2022
Table E: Effect of DR applied to different GAN models in the Single Image setting on DAVIS- YFCC100M.
DR X
Pixel f Diversity
Dist. to train
Method
SIFID
LPIPS
MS-SSIM L
0.13
0.26
0.69
0.54
0.24
SinGAN 0.15
0.15
0.26
0.68
0.55
0.26
0.09
0.29
0.65
0.59
0.25
ConSinGAN
0.05
0.11
0.30
0.63
0.60
0.28
0.13
0.18
0.77
0.49
0.08
FastGAN
0.15
0.14
0.21
0.74
0.52
0.13
0.05
0.04
0.95
0.33
0.06
SIV-GAN
0.15
0.08
0.33
0.63
0.66
0.31
The performance of the models with DR, as well our selected À values are included to Table F As seen from the table, DR plays the biggest role in combination with SIV-GAN, while applying DR to other methods leads only to minor improvement in diversity (0.01 0.03 MS-SSIM), For the single image GANs (Shaham et al. 2019: Hinz et al. 202T) this is explained by their multi-stage training schemes: their discriminators already overfit to all possible patches at given scale, the DR does not help to learn more diverse combinations. Single branch discriminator of the few-shot model (Liu et al. (02T)) overfits easily when trained with very little data, and DR alone cannot correct this. We conclude that our two-branch discriminator is crucial to leverage the benefit of the proposed diversity regularization.
C RESULTS ON FEW-SHOT IMAGE SYNTHESIS
C.I STANDARD FEW-SHOT IMAGE SYNTHESIS BENCHMARKS
In addition to the Single Image and Single Video settings, we conduct experiments also for few-shot image synthesis. Under this scenario, the task is to train a generative model on small dataset, typically containing up to several hundreds images. We experiment on the commonly used 100- shot datasets from (Zhao et al., 2020a) (Obama, Grumpy Cat, Panda) and 160- and 389-shot datasets from Si & Zhu (011) (Face-Cat, Face-Dog). These datasets have same image resolution of 256x256 and share common object-centric structure, depicting faces of humans or animals which are centred in all images.
Many methods address few-shot image synthesis in the paradigm of few-shot adaptation, adopting GAN models pre-trained on larger datasets (Wang et al., 2018: Mo et al.) 20200] Noguchi & Harada, 2019 Wang et al., 2020)) Another line of work proposes stabilize few-shot GAN training from scratch, by utilizing differentiable image augmentation (DA) and LeCam consistency regularization (CR) (Tseng et al. 202I), or by using a skip-layer excitation module (SLE) for faster learning of the generator with self-supervised discriminator (Liu et al., 2021)). The two most recent studies, LeCam CR (Tseng et al., 202T) and FastGAN (Liu et al.) 2021), demonstrated that is possible to outperform few-shot adaptation methods by using GAN models without pre- praininin achieving significantly better performance on the standard few-shot image synthesis benchmarks (see Table F). We select FastGAN (Liu et al. 2021) our main comparison model it is the state-of-the-art model with the official implementation code available in open source
To demonstrate the benefit of the SIV-GAN model design on the few-shot image synthesis task we apply the following modifications. Since in the few-shot setting the training set contains multiple data instances, we need to increase the capacity of our generator by increasing its depth by one ResNet block and increasing the channel multiplier (overall increasing to ~30M parameters from the original 5M). In order to enable a fast learning of the larger generator we adopt the SLE module of Liu et al. (2021). We make two changes to the configuration of our diseriminator. Firstly, to closer match the learning capacity of a more heavy-weight generator, we increase the width of our discriminator by doubling the channel multiplier in all layers. This step results in increase of trainable parameters count from 1.6M to 5M. Secondly, we move the branching point to a later stage, setting Nowwtteel This step is necessary to follow the structure of object-centric few-
19

---

Under review as a conference paper at ICLR 2022
Table F: Results on the few-shot image synthesis task. The FID is computed between 5000 gener- ated images and the whole training set. Bold and underlined indicate first and second best scores. Our SIV-GAN+ achieves better FID scores compared to FastGAN on all few-shot datasets, also outperforming the model from LeCam (Tseng et al. 2021) on four out of five datasets.
Dataset:
Obama
Grumpy Cat
Panda
Face Cat
Face Dog
FID
Number of images:
100
100
TO0
160
389
Scale/Shift (Noguchi & Harada
2019
O
50.72
34.20
21.38
54.83
83.04
MineGAN WWng et al.,
2020)
50.63
34.54
14.84
54.45
93.08
TransferGAN (Wang et al.
2018
48.73
34.06
23.20
52.61
82.38
FreezeD Mo et al. 2020)
41.87
31.22
17.95
47.70
70.46
LeCam (1 seng et al.
2021
33.16
24.93
10.16
34.18
54.88
FastGAN (iuu ct al.
2021)
38.59
27.08
9.63
33.50
53.39
SIV-GAN+
35.10
23.79
9.22
31.24
50.05
models which use pre-training
Table G: Results in the extremely low few-shot data setting, where the models are trained only on subsets of standard few-shot datasets. Our model outperforms the baseline in all data regimes both in quality and diversity, and does not suffer from training instabilities using only 25% of the training set. Collapsed runs with a high FII) for Fast-GAN are shown in red.
in extreme cases, such as
Panda (100)
Face Dog (389)
Method
100%
50%
25%
100%
50%
25%
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FastGAN
9.63
0.48
13.97
0.49
16.08
0.47
53.39
0.62
77.57
0.61
96.86
0.61
SIV-GAN+
9.22
0.51
11.57
0.51
9.58
0.50
50.05
0.63
64.21
0.62
62.61
0.62
shot datasets. As objects in such datasets cover whole images, we observed that the layout decision should be taken at more global scale (4x4 instead of the previous 32x32 spatial resolution for the layout representation), while the content representation should have large enough receptive field to see the complete object. Finally, in line with (Liu al.) (02T), we keep the self-suupriiiiio loss for discriminator, we found it to stabilize the training in the few-shot data regime. Further we refer to our extended model as SIV-GAN+
We train FastGAN and SIV-GAN+ for 100k iterations. We save the checkpoints every 10k iterations during training and report the best FID across all the checkpoints. The FID is computed between 5000 generated images and the whole reference dataset. The results shown in Table We gener- ally manage to reproduce the FID performance of FastGAN reported in (Liu et al.] 2021), T. obtaining slightly better results for all datasets. Table F shows that our SIV-GAN+ model outperforms Fast- GAN, achieving an average improvement of 8% in FID across all datasets. Moreover, SIV-GAN+ also shows better results than the recent GAN model of Tseng et al. (202T), yielding the best FID on four out of five commonly used few-shot image synthesis datasets. We show the visual com- parison our model with FastGAN in Fig. (two upper rows). Our model achieves a notably higher-quality synthesis on the Dog Face dataset, where FastGAN suffers from instabilities, having problems in generating good textures and correct shapes of dog faces.
C.2 EXTREMELY LOW FEW-SHOT SETTING
Training GANs in low data regimes is challenging, because a GAN discriminator is prone to overfit- ting and memorization effects, which can result in the divergence of the training progress. Existing state-of-the-art few-shot models (Tseng et al. 2021 Liu et al.) 202T were demonstrated to succeed in this task on datasets containing least 100 images (see App. C.I In this section, we investigate more extreme scenarios with datasets consisting of less than 100 images. We construct "extremely low" few-shot learning regimes by selecting subsets of Obama and Face-Dog datasets containing only 50% and 25% of original training images.
The quantitative results for the extremely low few-shot settings are shown in Table G while the visual results on 25%-subsets is shown in Fig. E (two bottom rows). As seen from the figure, the
20

---

Under review as a conference paper at ICLR 2022
FastGAN, trained on full dataset (389 images)
SIV-GAN+ trained on full dataset (389 images)
FastGAN, trained % of data (98 images)
SIV-GAN+, trained on 25% of data (98 images)
Figure E: Visual results in the few-shot setting on the Face Dog dataset. The two upper blocks show generated images for the models trained the full dataset (389 images), while the bottom blocks show the results from training only using 25% of data (98 images). SIV-GAN+ demonstrates only superior performance on the standard few-shot setting, but also successfully deals with an extreme few-shot scenario (25% of data), where FastGAN has a significant drop in performance.
FastGAN model experiences drop in performance in the more extreme data setting, when using less than 100 images for training, producing unrealistic textures or incoherent shapes of dog faces. In Table G this effect reflected in high FID scores (highlighted in red). In contrast, SIV-GAN+ shows good image quality uniformly across settings, reaching good FID even training sets con- taining very few images. Importantly, the gain in FID is achieved at the cost of diversity: our model has higher LPIPS diversity score in all data regimes compared to the baseline. Moreover, Table Gshows similar diversity scores for SIV-GAN+ in various settings, indicating that our model can be successfully scaled to various extremely low-data regimes maintaining similar levels of syn- thesis diversity. Overall, the results demonstrate that our proposed two-branch discriminator helps to mitigate discriminator overfitting in few-shot regimes, stabilizing the training, and thus helping to maintain good quality and diversity even when the model is trained on datasets with very few images.
21

---

Under review as a conference paper at ICLR 2022
SIV-GAN+ generated images
Nearest training frames
SIV-GAN+ generated images
Nearest training frames
SIV-GAN+ generated images
FastGAN generated images
Figure F: Visual examples in the fhw-shot setting on the ADE-Outdoors and LSUN Church datasets. Despite being trained only 100 images, SIV-GAN+ does not suffer from memorization issues and achieves an astonishing capability to combine objects from different training images, preserving the global coherency of scenes. contrast, FastGAN does not achieve high-quality synthesis, struggling to generate coherent images with fine textures.
Table H: Results few-shot datasets with a non-object-centrie structure. Our model outperforms
FastGAN both in quality and diversity on both datasets.
ADE-Outdoor (100)
LSUN Church (100)
Method
100%
50%
25%
100%
50%
25%
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FastGAN
113.2
0.62
125.1
0.62
98.28
0.61
90.65
0.58
121.4
0.58
114.1
0.56
SIV-GAN+
88.46
0.64
82.89
0.63
52.96
0.63
85.01
0.60
103.8
0.60
90.10
0.59
C.3 NON-OBJECT-CENTRCG FEW-SHOT IMAGE SYNTHESIS
As mentioned in App. C.I. standard few-shot image synthesis benchmarks include only object- centric datasets, such as centred faces of people or animals. Such datasets have very limited vari- ability in layouts, having very similar locations of face parts in all images. In this section, explore few-shot image synthesis on datasets with more complex structures. For this, we construct subsets of the ADE-Outdoors (Zhou al. 2017b) and LSUN-Church YYu et al. 2015) datasets, consisting of 100 images with resolution 256x Such datasets form a more difficult task, because the mod- els have to leam how to combine objects with different semantics from different images preserving the global scene layout.
The quantitative comparison between SIV-GAN+ and FastGAN shown Table H Our model outperforms FasIGAN in different data regimes (100, 50, 25 images) in both quality and diversity. Notably, the structure of the ADE-Oure Outdoors and LSUN-Churd datasets provides potential to gen- erate novel compositions of objects which not seen in the training data. We show examples of
22

---

Under review as a conference paper at ICLR 2022
interesting scene compositions produced by model in Fig. Being trained only 100 images of outdoor landscapes or churches, SIV-GAN+ does not memorize the training examples, generat- ing novel compositions of mountains exchanging towers of different churches. In the meantime, FastGAN suffers from instabilities in this setting, struggling to generate images with fine textures.
D ADDITIONAL QUALITATIVE AND QUANTITATIVE RESULTS
D.1 QUALITATIVE RESULTS IN THE SINGLE IMAGE AND THE SINGLE VIDEO SETTINGS
In Fig. G Fig. H and Fig. we present additional qualitative results. In Fig. G and Fig. H we compare SIV-GAN to other models in the Single Image setting on the Places dataset Zhou et al. 2017a). We note that the previous single-image methods, SinGAN and ConSinGAN, tend to shuffle image patches in a globally semantic regions. Moreover, such methods do not preserve the appearance of objects, for example, by washing away sculptures in the garden or a man the forest. For images with complex scenes these methods do provide noticeable diversity, as in the example with children playground. On the other hand, FastGNN (Liu et al. 2021) (see Fig. H suffers from memorization, falling into reproducing training image or its flipped version. In contrast, our SIV-GAN preserves objects, maintains global layout coherency and produces images with significant variability.
way, as object textures may leak into surfaces of other
Fig. shows the images generated by SIV-GAN the Single Video setting. Compared to the Single Image setting, in this scenario there is much more data learn from, and generative models can learn more interesting combinations of objects in the scenes. As seen from the figure, our SIV-GAN manages to preserve the context of the training frames, the same time adding non-trivial semantic changes to original scenes. For example, for the shown videos, the generated frames can have a different number of balloons in sky, and combinations of planes, boats or buildings that were not seen during training.
D.2 HIGH RESOLUTION IMAGE SYNTHESIS RESULTS
The experiments described in Sec. 4 were conducted at the image resolution of 192x320. In this section, demonstrate the ability of our model to gener- ate images at higher resolution 512x896 on the MAVIS-YFCCIOOM dataset. For this, add one ResNet block to the generator and discriminator, and change the input noise shape from 3x5 to 4x7 (see App. D for more architectural details). After this change, the model produces images much higher image resolution of 896x512. We show the visual results for high resolution image synthesis of SIV- GAN Fig. J We don't observe any issues caused by the change of image resolution. As shown in Table the performance of the model is similar at different scales: SIFID of 0.08 and LPIPS of 0.29 at resolution 512x896 is aligned well with quality and diversity at resolution 192x320 (0.08 and 0.33 in Table T).
Table I: Comparison of synthesis quality and diversity different image resolutions on DAVIS-YFCCIOOM in the Single Image set- ting.
Image resolution SIFID & LPIPS 1
192x320
0.08
0.33
512x896
0.08
0.29
23

---

Under review as a conference paper at ICLR 2022
SinGAN
ConSinGAN
SIV-GAN
Training image
SinGAN
ConSinGAN
SIV-GAN
Training image
SinGAN
ConSinGAN
SIV-GAN
Training image
SinGAN
ConSinGAN
SIV-GAN
Training image
SinGAN
ConSinGAN
SIV-GAN
Training image
Figure G: Comparison with other methods in the Single Image setting on Places. Single-image GANs of (Shaham et al. 2019 Hinz et al. 2021) are prone to incoherently shuffle image patches. In contrast, SIV-GAN produces images preserving the appearance of objects.
24

---

Under review as a conference paper at ICLR 2022
SinGAN
FastGAN
SIV-GAN
Training image
SinGAN
FastGAN
SIV-GAN
Training image
SinGAN
FastGAN
SIV-GAN
Training image
SinGAN
FastGAN
SIV-GAN
Training image
SinGAN
FastGAN
SIV-GAN
Training image
Figure H: Comparison with other methods in the Single Image setting Places. The single-image GAN of (Shaham et al., 2019) prone to incoherently shuffle image patches, and the few-shot FastGAN model (Liu et al.; 202T) reproduces the training image its flipped version. In contrast, SIV-GAN produces diverse images, preserving the appearance of objects.
25

---

Under review as a conference paper at ICLR 2022
Figure I: Additional qualitative results in the Single Video setting. The training sequences are shown in grey frames. Given single video for training, our SIV- GIV-GAN produces images that different to the training frames. For example, for video with air balloons, the generated images have different number balloons, while for video with a boat, the model generates novel combinations boats and buildings not seen during training.
26

---

Under review as a conference paper at ICLR 2022
Generated samples from single training frame
Training image
Generated samples from a single training frame
Training image
Generated samples from single training frame
Training image
Figure J: SIV-GAN results at high image resolution of 512x896 in the Single Image setting. Our model shows good scalability different resolutions, maintaining the quality and diversity of scene compositions. For example, given only one image with a bike rider, SIV-GAN can produce image with two bikers, without the biker, as well as change the bike lane path. For the frame with a parkour jumper, the scene can be resynthesized without the jumper, with different positioning of barriers. Finally, for the image with a breakdancer, the number and positions of the dancer and observers can be changed.
27

---

Under review as a conference paper at ICLR 2022
D.3 SIFID USING DIFFERENT INCEPTIONVS 3 LAYERS
To evaluate image quality different scales, we additionally compute the SIFID metric (Shaham it al. 2019) at various Inception V3 layers. Its original formulation uses Inception V3 features before the first pooling layer, at a spatial resolution of HxW, Such metric captures only low-level image details, such as colors and textures, and is not able to capture high-level semantic image properties, such as appearance of objects or global layouts. Therefore, to evaluate realism at different scales, we additionally use features before the second pooling layer (at a resolution of # ) pre-classifier features ( (HxW) and the final Inception features (1 X 1), that are commonly used to compute FID (Heusel et al. 2017). To obtain the SIFID score in the Single Image setting, we generate 100 images and then compute their mean SIFID the training image (corresponding columns mean in Table []. The results on the DAVIS-YFCCION CFCC100M dataset (Perazzi et al., 2016 Thomee et al. 2016) for the Single Image and Single Video settings are presented in Table
We observe no disagreement with the metrics reported in the main paper. SIV-GAN achieves bet- ter image quality than the comparison models in both settings, as measured at all the InceptionV3 layers. Notably, parable low-level SIFID (at a resolution of HxW, ), but achieve poor high-level scores ( (HXW 1 1). This indicates that these models successfully learn textures of given images, but fail to reproduce scenes at a mid-level (objects) and the global scale (layout). In contrast, our proposed two-branch discriminator allows learning the scene appearance at all scales, enabling generation of plausible images not only with fine textures, but also with correct appearance of objects and globally-coherent layouts.
that single-iman -iaeee methods, SinGAN and ConSinGAN, achieve com-
16
Table J: Comparison of SIFID at different scales on Single Video settings.
in the Single Image and
Single Image
Single Video
HxW
HxW
HxW
1 X
HxW
HxW
HxW 16
1x]
1
16
SinGAN
0.13
4.93
34.52
2510
2.47
13.61
96.35
411
ConSinGAN
0.09
2.94
27.33
1960
2.74
14.73
74.50
392
FasIGAN
0.13
2.89
19.48
1340
0.79
1.75
9.24
141
SIV-GAN
0.08
1.29
16.30
1100
0.55
1.32
5.14
115
D.4 COMPARISON OF SYNTHESIS DIVERSITY IN THE SINGLE IMAGE AND SINGLE VIDEO SETTINGS
In Table K we compare the diversity among the images generated from a single image and from single video on the DAVIS-YFCCIONS dataset. In the Single Image setting we use only one frame in the middle of the sequence for training, while in the Single Video setting we use all video frames as training data. As seen from the table, the diversity of generated samples is notably higher for the case when all frames of a video
used (0.43 against 0.33 LPIPS). The visual difference between the settings is illustrated in Fig. K The model, trained only on the middle frame, produces slight variations of the training image. For example, such model changes the number of windows in the building or modifies the geometry of the concrete barrier. On the other hand, the model, trained on full video, achieves more complex transformations, being capable of changing the layout of buildings or removing person from the scene.
In practice, capturing a short video clip can take almost as little effort as collecting an image. As we show in our experiments, using short videos for training allows to generate significantly more diverse images, compared to the case when only a single image is used. This way, we would like to
Table K: Comparison of the diversity among the images generated in the Single Image and the Single Video settings on the DAVIS- YFCC100M dataset.
Data setting
LPIPS +
MS-SSIM 1
Single Image
0.33
0.63
Single Video
0.43
0.54
28

---

Under review as a conference paper at ICLR 2022
Generated images from the single frame
Video frame
Generated images from the whole video
Figure K: Difference between models trained on single middle frame of video and on the full video sequence. the Single Image setting, the model generates only slight geometric transfor- mations, such as varying the number of windows concrete barrier. In contrast, the Single Video setting allows to provide more substantial diversity, removing the person from the frame or changing the layout of buildings.
a building, or modifying the geometry of
draw the attention of the community to the Single Video setting, which we believe to be helpful in extending the usability of generative models for practical applications.
D.5 QUALITATIVE EFFECT OF THE PROPOSED DIVERSITY REGULARIZATION AND FEATURE AUGMENTATION
The proposed diversity regularization (DR) and feature augmentation (FA) are essential components for SIV-GAN to achieve a high diversity of synthesis. Table 3 demonstrates a quantitative effect of these two components
the performance of our model. In Fig. L we supplement this analysis with a visual study, showing the images generated by our full model, as well as by the models trained without DR or FA in the Single Image setting.
We observe that the visual results correspond well to the numbers from Table 3 As seen from Fig. D SIV-GAN without DR does not manage to mitigate overfitting, suffering from mode collapse. Such model learns to memorize the original training sample and thus reproduces it without any modifications. In Table 3 this corresponds to the low LPIPS and Dist. to train scores of 0.04 and 0.06. The model with DR but without FA manages to achieve diverse image synthesis. However, such model produces only modest diversity in content and layouts. For example, for an image with waves and surfers, it does not change the number of surfers, while for images with rocks, it typically translates rocks to new locations but does not change their shape. In Table 3 this is reflected in the increased LPIPS and Dist. to train scores of 0.27 and 0.33. Finally, our full model, trained with both DR and FA, enables generating more interesting novel scene compositions, varying global scene layouts and changing the content distribution. For instance, it can generate different number of surfers or mountains compared to the training sample, or modify the shapes of rocks. Correspondingly. in Table 3 our full model shows a much higher diversity, achieving the highest LPIPS of 0..3 and Dist. to train of 0.37.
E DETAILED DESCRIPTION OF THE ARCHITECTURE AND TRAINING DETAILS
E ANALYSIS OF STRATEGIES TO GENERATE IMAGES OF RECTANGULAR SHAPES
Commonly used state-of-the-art GAN models, such as (Brock et al. 2019 Karnewar & Wang 2020 generate images of square shapes. In our studied settings, single images video frames can be rectangular, having resolutions of different horizontal and vertical aspect ratios. In contrast, we design our model to be compatible with images of different resolutions. In this section, we analyse two possible strategies to achieve this.
One possible strategy is to follow the design of SinGAN (Shaham et al. 2019). which has several training stages. Such model proposes to reach the final resolution following geometric progression, where the image is upsampled after each training stage. In this scheme, the progression multiplier is computed based on the image size, the number of stages, and predefined image size first model stage. Typically, this multiplier is non-integer and different for horizontal and vertical axes. For SIV-GAN, we adopted such scheme by progressively upsampling the image after each generator
29

---

Under review as a conference paper at ICLR 2022
SIV-GAN without DR
SIV-GAN without FA
SIV-GAN
Training image
SIV-GAN without DR
SIV-GAN without FA
SIV-GAN
Training image
SIV-GAN without DR
SIV-GAN without FA
SIV-GAN
Training image
Figure L: Qualitative ablation on the proposed diversity regularization and feature augmentation. Without DR, the model does not mitigate memorization, producing only images that are perceptually indistinguishable from the original sample. Without FA, SIV-GAN achieves only modest diversity in content and layouts. Finally, using both DR and FA enables generating more interesting novel scene compositions, varying global scene layouts, or duplicating and removing objects.
block, starting from noise of shape (4 x 4) and reaching the exact image shape by the final generator layers. our experiments, such upsampling scheme leads texture artefacts generated images (see Fig. M). We observed that the artefacts are caused by non-integer interpolation between layers: backpropagation through such transformation causes "grid" effect, which hard to correct during training of the generator. The difference in the texture quality in comparison to (Sahaa et al. (2019); Hinz et al. (202I) explained by the fact that these models are trained in multiple stages, without requiring backpropagation through upsampling layers. In contrast, SIV-GAN is single stage model, and gradients cannot bypass the interpolation.
To avoid corruptions of textures, we therefore select an alternative solution. As discussed in App. we propose to follow the architecture of Brock et al. (2019), employing x2 upsampling after each residual block, modifying the shape of input noise, adjusting to fit better the final image
Table L: Effect of different strategies to match the rectangular shape of the original training image on the DAVIS-YFCCIOOM dataset.
Single Image
Single Video SIFID J LPIPS
Interpolation
SIFID 4 LPIPS 1
Progressive upsampling
0.11
0.32
0.65
0.42
Adjusted noise
0.08
0.33
0.55
0.43
30

---

Under review as a conference paper at ICLR 2022
multiple resolution. Note that for this approach the final resolution of generated images is of the imput noise shape, which can sometimes not precisely fit to the resolution of an original image (for example, in case when the training image has odd number of pixels in one of the dimensions). Nevertheless, with such approach the shape of an image is preserved better, compared to generating square images as in (Brock al. 2019; Karnewar & Wang (020). As seen from Fig. M adjusting the noise shape leads to better quality of images, overcoming the interpolation problem.
Table conffrrm our analysis. We observe that using the progressive upsampling, as in Sha- ham et al. (2019), leads to decreased quality of images, as measured by SIFID. Moreover, such approach produces slightly lower diver- sity. Therefore, for our final model design, choose to adjust input noise shape, as it leads to better performance.
Progressive upsampling
Training image
Adjusted noise
E.2 ARCHITECTURE
AND TRAINING DETAILS
Architecture details of our SIV-GAN are sum- marized Tables M and N Our model has generator G' and discriminator D consist- ing of three parts: low-level feature extractor Dtow-levels layout branch Dlayout, and con- tent branch Deontent- Our design mainly fol- lows the one of BigGAN (Brock al.) 2019), applying ResNet blocks at both the generator and the discriminator. Following MSG-GAN
Figure M: Different strategies to achieve rectan- gular shapes. Using progressive upsampling, as in (Shaham et al. 2019), leads to artifacts in textures (e.g. chequered "waves" on the rock). Controlling output image resolution through adjusting noise shape helps to overcome this issue and generate textures of good quality.
(Karnewar& & Wang 2020), we use skip-connections between G and D. We thus generate images differrntt resolutions and pass them discriminator at different scales. In contrast (Karnewar & Wang 2020), our discriminator is not symmetric to generator, as after Now discriminator blocks we apply branching. Therefore, we generate images only at Noow-teeel highest resolutions, and then pass them at each block of Diow-tevel- To incorporate generated images at intermediate D blocks, we apply a 1 x convolution, and concatenate the obtained tensor to the current D features (strategy Piincaat from (Karnewar & Wang )020
The branches Diayout and Deontent are formed by usual D ResNet blocks, with two modifications. First, the content branch has 1 spatial dimensions, the used convolutions have kernel size (1 x 1) and are technically equivalent to a fully fulneccon layer. Secondly, the layout branch has channel, so its convolutions always have the number of channels equal to 1. To form binary real-fake decision tensor, after each D block we use 1 x 1 convolution, which maps the current features to one-channel map with logits.
In order to fit better the training image size, we change the resolution of the input noise. We keep the noise resolution around (4 x 4), adjusting it to different ratios to fit shapes of training images closer. For example, the input noise in experiments
resized to (3 x 4), (3 5) (4 x 4), depending on the final image resolution. For evaluation, we bilinearly resized generated images to the resolution of original training image. Tables M and N present an example for input noise of resolution 3 x 5, which correspond to the frames from the DAVIS video dataset (Perazzi et al., 2016).
We train model with ADAM (Kingma & Ba 2015) optimizer, using batch size of 5, momen- tums (31, 32) = (0.5, 0.999), learning rates 0.0002 for both the generator and the discriminator, All experiments were conducted on single GTX 1080 GPU with 12 GB RAM memory.
31

---

Under review as a conference paper at ICLR 2022
Table M: The SIV-GAN generator. In this example, the configuration is presented for the input noise of size (3 x 5) and the final resolution of (192 320), corresponding training the DAVIS- YFCCI00M dataset in the Single Video setting.
Operation
Input Size
Output
Size
ConvTrrans2DD
z (64,1,1)
up_0
(256,3,5)
ResBlock-Up
up_0 (256,3,5)
up-1
(256,3,5)
ResBlock-Up
up (256,3,5)
up 2
(256,6,10)
ResBlock-Up
up-2 (256,6,10)
up_2
(256,12,20)
ResBlock-Up
up_3
(256,
up_3
(256,24,40)
ResBlock- Up
up_4 (256,24,40)
up_4
(256,48.80)
ResBlock-Up
up_5
(256,48,80)
up_5
(128,96, 160)
ResBlock-Up
up__ (128,96, 160)
up_6
(64.192,320)
Conv2D, TanH
up 5 (128,48,80)
image
(3,48,80)
Conv2D, TanH
up 6 (64.9,1160)
image
(3,96.160)
Conv2D, TanH
up_7 (32.192.320)
image_0 (3,192,320)
Table N: The SIV-GAN discriminator. In this example, the configuration is presented for the input noise of size (3 x 5) and the final resolution of (192 320), corresponding to training the DAVIS- YFCC100M dataset in the Single Video setting.
Operation
Input
Size
Output
Size
Low-level features Diow level
Conv2D
image_0 (3,192,32))
feat_0 (32,192,320)
Conv2D
image_1 (3,96,100)
feat_i 160)
Conv2D
image_2 (3,48,80)
feat_2 (16,48,80)
ResBlock-Down
feat_0 (32,
down_0 (64.96,160)
down_0 (64,96,160)
ResBlock-Down
down_1 (128,48,80)
feat_l
(8, 96, 160)
down
(128,48,80)
ResBlock-Down
(256.24.40)
feat_2
(16,48, 80)
Content branch D Donntent
AvgPool
F
(256,24,40)
c.con (256,1,1)
ResBlock-Down F.con 1256,1,1)
cont_0 (256,1,1)
ResBlock Down cont (256,1,1)
cont (256,1,1)
ResBlock-Dovd Down cont_1 (256,1,1)
cont_2 (256,1,1)
ResBlock BesBlock-Down cont.2
(256,1,1)
cont.3 (256,1,1)
Layout branch Diayout
Conv2D
F
(256,24,40)
lay (1,24,40)
ResBlock- -own F_lay
(1,24,40)
lay_0
(1,12,20)
ResBlock-Dord -ownn lay 0
(1,12,20)
lay (1,6,10)
ResBlock-Dond Down lay_1
(1,6,10)
lay_2 (1,3,5)
ResBlock-Down lay-2
(1,3,5)
lay_3
(1,3,5)
32


Reviews:
[{"review_id": "z_IES6HoIBG", "reviewer": "Reviewer_TwJf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method generates good quality results. However, I think the technical novelty is limited and the task of learning from a single video is much simpler than learning from a single image.\n\n1. The method proposes to learn new scene generation from a short video. Although a short video takes similar access effort as an image, it literally contains at least tens or hundreds images. This task is much easier than learning from a single image. \n\n2.The paper didn't use the temporal information in the video, so what's the difference from a video and hundreds of frames? Also, hundreds of images are often enough for image-to-image translation, e.g. [A] uses hundreds of  frames to learn a render-to-realistic GAN.\n\n[A] Yi et al. Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose. Arxiv.\n\n3. The proposed two-branch discriminator assumed content information is spatial independent, which is strange since spatial distribution also provides much content information. And the method mainly aggregates spatial information and channel information separately, which has also been proposed in previous methods, e.g. [B].\n\n[B] Fu et al. Dual Attention Network for Scene Segmentation. CVPR 2019.\n\n4. The diversity loss is also quite simple, i.e. given two latent codes z1, z2, encourage the generated images G(z1), G(z2) to be different. Shouldn't the distance between two latent codes affect the distance between generated images? In current setting, if z1 and z2 is very close, the loss also encourages the generated images G(z1), G(z2) to be different.", "summary_of_the_paper": "The paper proposed a new method for generating new scene compositions using a single video. The method proposes a two-branch discriminator, one for content discrimination and the other for layout discrimination. It proposes a diversity loss for generator to encourage images generated from different latent codes to be perceptually different. Experiments show the method generates diverse and high-quality scene images.", "summary_of_the_review": "In summary, the statement of learning from a single video is basically the same as learning from hundreds of images, and is thus much easier than learning from a single image. The proposed contributions, i.e., the two-branch discriminator and diversity loss, lack technical novelty (see above). So I think the paper is not good enough for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uDeCx56gj6b", "reviewer": "Reviewer_LNdN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduces a new GAN architecture for unconditional image synthesis after being trained on a single image/video. Based on the observation that previous models either fail to generate realistic layouts (SinGAN, ConSinGAN) or collapse to the training data without diversity (FastGAN) the paper proposes a new discriminator architecture to address this.\nAfter a couple of layers to extract image features, the discriminator splits into two independent branches. One branch is trained to judge the global image layout while the second branch is trained to evaluate individual object quality. Aditionally, a feature augmentation is introduced to further increase the diversity.\nAt the generator side, a diversity regularization is added which forces to generator to generate different images for different latent samples.\n\nThe resulting model outperforms all baselines when comparing the image quality and diversity. Compared to single-image GANs it obtains a much better quality by generating realistic layouts and compared to few-shot GANs it obtains a much higher diversity. A detailed ablation study shows the effects and improvements of the individual novelties.\n\nWhile the approach is intuitive and does seem to lead to clear improvements I wonder how well it translates to applications such as image harmonization/editing/animation/retargeting/etc. While unconditional synthesis is interesting I think many of the more obvious direct applications do not involve pure unconditional generation. Since at least the single-image GANs seem to be able to perform those tasks without or only minor changes in the training pipeline it should be relatively simple to evaluate this approach on some of these tasks, too.\n\nRelated to that: while the diversity regularization in general makes sense I wonder if really forcing the features to be different regardless of the distance in z is the best way of doing. This may affect tasks such as animation or latent space interpolation. Have you tried the \"normal\" way of just forcing the features to be different dependent on the distance in z? I don't see a compelling reason why this shouldn't work, too.\n\nAlso, it would be helpful to have a user study to compare your approach with the baselines. Since the main metrics (to a degree) oppose each other (e.g. SIFID vs diversity), having humans compare the models directly would be helpful. You could either reproduce the user studies from SinGAN/ConSinGAN (compare against training image) or compare the models against each other directly.\n\nFinally, do you have any numbers on the model size compared to the baselines, training time/effort, etc? It seems like your model is trained for many more iterations than SinGAN/ConSinGAN and not in a progressive manner, i.e. I assume it takes longer to train your model?\n\n", "summary_of_the_paper": "This paper proposes a single-image/single-vide GAN model that obtains both high quality and high diversity in the generated images, as opposed to previous models that only achieve one of the two. To achieve this, the discriminator consists of two branches that evaluate global layout and quality independently and a novel regularization approach for the generator is introduced. Comparisons to SinGAN, ConSinGAN, and FastGAN show improved performance.", "summary_of_the_review": "The proposed novleties make sense and seem to lead to clear improvements. However, the evaluation is only based on unconditional image synthesis and ignores many more practical applications for single-image models such as animation, harmonization, retargeting, etc. A more broad evaluation on some of these tasks coupled with a user study would be more appropriate.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gPf18gXb8uT", "reviewer": "Reviewer_DXh5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " \n##########################################################################\n\nPros:  \n1. The paper overall is well-written and easy to follow. \n2. The authors have conducted extensive experiments (quantitative and qualitative) in the paper. \n \n##########################################################################\n \nCons: \n1. [Motivation] The authors worried about training GANs with limited data will lead to memorization of training images (claimed in the abstract). However, the results shown in Figure 1 look like a kind of repetition of patches of the training samples, which conflicts with the motivation. \n \n2. [Motivation] The application of training from a single sample (in the first paragraph on Page 2) is not convincing. Are there any other practical examples to show the practical value of learning from a single sample?\n \n3. [Motivation] The first contribution of learning from a single video is not convincing. Similar to generating a single image by learning from a single image, readers typically expect learning to generate a video by learning from a single video. The authors should provide video results that show temporal coherence. Otherwise, learning from a single video is actually learning from multiple frames.\n \n4. [Approach] Two-branch-like discriminators have been widely explored in GAN. For example, local and global discriminators (Globally and locally consistent image completion. Iizuka et al., TPOG 2017). I think the novelty can be limited. \n \n5. [Approach] Since there is only one real sample during training, I don’t think this two-branch discriminator is able to prevent memorization as claimed on Page 5. I think the effective technique is feature augmentation. Please show the results without feature augmentation and regularization to verify the effectiveness of the discriminator.\n \n6. [Approach] I have trouble understanding the regularization technique in Section 3.2. What does “be more or less equally different from each other” mean? In my view, the regularization technique is actually a maximization of the feature distance of different samples. \n \n7. [Experiments] The results in Figure A is confusing. The results of ”no content branch” also show a worse layout, which is hard to verify the effectiveness of the layout branch and content branch respectively. \n \n8. [Experiments] I suggest ablation studies (by visual comparisons) on the feature augmentation and the diversity regularization, to verify the effectiveness of the proposed model.\n", "summary_of_the_paper": "The paper presents a new GAN model that generates new scene compositions from a single training image or a single video clip.  This is achieved by a two-branches (content-layout) discriminator and a diversity regularization technique. The paper also claims as the first work learning from a single video.", "summary_of_the_review": " \nOverall, I vote for rejecting. I am especially concerned about the novelty of the two-branch-like discriminator, the effectiveness of the discriminator, and the motivation of learning from a single video. Please address the concerns listed in Cons during the rebuttal period.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PMSESX_ep-t", "reviewer": "Reviewer_inBU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper improve the sinGAN on image generation using very few training images. Technically speaking, the proposed solution is interesting, but the technical contribution is not super significant. It is not new to disentangle the image into content and layout parts by restricting the feature dimension -- deep vector for content without spatial information and shallow map for layout without semantic information, similar to the papers such as Swapping Autoencoder. The results look better than sinGAN, but still suffer from large duplicated content and distortion artifacts, but that's understandable for such an extremely challenging problem.\n\nFor the technical details, the paper is written well and easy to understand. I just have a few small questions and comments:\n1. In feature augmentation, what exactly does \"For the single image setting this is done by mixing the features of two different augmentations of the original image\"? What is the different augmentation here?\n2. In feature augmentation again, I am a bit concerned about the proposed augmentation. Take layout as an example, to crop a rectangle of feature to another image does not make a reasonable real image in my understanding, which is equal to crop a random patch from one frame and paste to another frame, the composted image will look strange. I am either not sure about the content augmentation. A discussion could help understand the intuition.\n3. No visual comparisons for many ablation analysis, including DR and FA.\n4. Would it make sense to add another reconstruction loss by combining content and layout features through another decoder to form an autoencoder? It might help enforce feature completeness and make training easier? But I'm not sure.\n5. Are those editing applications in sinGAN (Fig.12) supported?", "summary_of_the_paper": "This paper proposes an approach to train an image generator using only a short video clip or a single image. The overall framework follows the design of sinGAN that has an unconditional generator that generates new images from random noises, and the generated images and the input real images/frames are fed to discriminators to enforce similar distributions. The key contribution is to disentangle the image into content and layout features, and use different discriminators, which is claimed to reduce the memorization issue.", "summary_of_the_review": "Overall I like this paper, which is solid, complete, with sufficient experiments and evaluations. I am okay with acceptance, but I am a bit concerned that the contribution might not be significant enough for acceptance, since it is essentially the sinGAN setting plus content/layout separation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors consider the problem of unconditional image generation in the low-data regime, such as learning from frames of a single video or even from a single image. The main idea is to apply GANs with a specific two-branch discriminator architecture such that the content features and layout features are handled independently. Secondly, to improve the variability of generated images the authors apply diversity regularization. The authors show that the proposed model is able to, to a certain extent, generate diverse high quality samples.

The paper is well-written, the authors described their method and the evaluation protocol thoroughly and clearly. The reviewers felt that this submission was borderline, with questionable novelty and significance. In an extensive rebuttal and discussion phase the authors addressed several raised challenges and improved their paper. However, two points remain:
- **Technical novelty**: content and layout separation as well as diversity regularisation previously appeared in many contexts and papers.
- **Motivation and practicality**: One of the main arguments for the utility of the proposed method is to use it for data augmentation. While it may indeed result in content-based augmentations, it nevertheless necessitates training of a GAN for every single image, which is severely limiting in practice.

After reading the manuscript, reviews, and the rebuttals, my view is that the paper is below the acceptance bar and I agree with the points on novelty and significance. In particular, the main application to data augmentation seems to be "unexciting" and the proposed method impractical. At the same time the proposed method is a combination of already known techniques, albeit in a different setting. I suggest the authors condense the arguments in the extensive rebuttal to improve the points raised above and resubmit.
Decision: Reject