Generating Novel Scene Compositions from Single Images and Videos

1 INTRODUCTION . The quality of synthetic images produced by generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) has greatly improved in recent years ( Zhang et al. , 2019 ; Brock et al. , 2019 ; Park et al. , 2019 ; Karras et al. , 2019 ; Lin et al. , 2019 ; Schönfeld et al. , 2020 ; Karras et al. , 2020b ) . These impressive results are in large part enabled by the availability of large , diverse datasets , typically consisting of tens of thousands of images . This dependency on the availability of training data limits the applicability of GANs in domains where collecting a large dataset is not feasible . In some realworld applications , collection of even a small dataset remains challenging due to specific constraints related to privacy , copyright status , subject type , geographical location , time , and dangerous or hazardous environments . It may happen that rare objects or events are present only in one image or video , and it is difficult to obtain a second one . For example , this includes images of exclusive artworks or videos of traffic accidents recorded in extreme conditions . Enabling GANs to learn in the extremely low data regimes , such as learning from the single data instance examples , thus has the potential to improve their utilization in practice . Prior work ( Shocher et al. , 2019 ; Shaham et al. , 2019 ; Hinz et al. , 2021 ) focused only on learning generative models from a single image ( Fig . 1 , rows 3-5 ) . In this work , we introduce a new task of learning to synthesize images from a single video ( Fig . 1 , rows 1-2 ) , where the training data is a collection of frames from a 2-10 second video clip . In practice , capturing such a short video takes almost as little effort as collecting a single image . However , compared to a single image , a video contains much more information about the scene and the objects of interest ( e.g. , different poses and locations of objects , various camera views , dynamic backgrounds ) . Learning from a video can enable synthesis of higher quality , more diverse images , while still keeping very low requirements on data availability , and therefore can improve GAN usability for practical applications . Training GANs in such extremely low data regimes is challenging as the models are prone to overfitting , leading to replication of training data or poor synthesis quality due to training instability ( Karras et al. , 2020a ) . For example , applying the few-shot image synthesis models , such as FastGAN ( Liu et al. , 2021 ) , to learn from a single image or video leads to severe memorization problems ( see Sec . 4 ) . To mitigate memorization , single-image GAN models ( Shaham et al. , 2019 ; Hinz et al. , 2021 ) proposed to learn an internal patch-based distribution of an image , employing a cascade of multiscale patch-GANs ( Isola et al. , 2017 ) trained in multiple stages . Though these models overcome memorization , producing different versions of a training image , they can not learn high-level semantic properties of the scene , e.g. , to dissect objects from the background . They often suffer from incoherent shuffling of image patches , distorting objects and producing unrealistic layouts ( see Fig . 4 and 5 ) . Furthermore , they have more difficulty to generate realistic scenes from a single video . We aim to go beyond patch-based learning , seeking to generate novel , plausible compositions of objects in the scene , while preserving the original image context . We want to learn a model which is able to compose new layouts , re-arrange objects in the scene , remove or duplicate instances , and change their shape and size . The new scene compositions should be visually plausible , with objects preserving their appearance and natural shape , and scene layouts looking realistic to the human eye . Reaching such synthesis quality from a single image or a single video is notably challenging , as the model needs to learn to distinguish objects and background using only one image or few training frames , without any direct object supervision ( e.g. , mask annotations ) . To this end , we introduce SIV-GAN ( Single Image and Video GAN ) , an unconditional , one-stage GAN model , capable of learning from the single data instances to generate images that are substantially different from the original training sample , while still preserving its context . This is achieved by two key ingredients : the novel discriminator design and the proposed diversity regularization for the generator . Our discriminator has two branches , separately judging image content and scene layout realism . The content branch evaluates objects ’ fidelity irrespective of their spatial arrangement , while the layout branch looks only at the global scene coherency . Disentangling the discriminator ’ s decision about content and layout helps to prevent overfitting and provides a more informative signal to the generator . To achieve a high diversity among generated samples , we further extend the regularization technique of ( Yang et al. , 2019 ) to unconditional image synthesis in single data instance regimes . The prior work of ( Yang et al. , 2019 ) encourages the generation of different images depending on their input latent codes , thus the difference between images is proportional to the distance between their codes in the latent space . Assuming that in case of a single image or a single video all generated images should belong to one semantic domain ( i.e . preserve the original training sample context ) , and thus should be more or less equally different from each other , we apply diversity regularization uniformly , independent of the latent space distance . Moreover , we use the regularization in the feature space , inducing both high- and low-level diversity . We demonstrate the effectiveness of our model for the single image setting as well as for the novel single video setting in Sec . 4 . SIV-GAN is the first model that successfully learns in both of these extremely low-data settings , improving over prior work ( Shaham et al. , 2019 ; Hinz et al. , 2021 ; Liu et al. , 2021 ) in both image quality and diversity . In contrast to FastGAN ( Liu et al. , 2021 ) , our model does not suffer from memorization ( see Table 1 and 2 ) ; while compared to single-image GANs ( Shaham et al. , 2019 ; Hinz et al. , 2021 ) , our model does not distort objects , preserving their appearance . In summary , our contributions are : 1 ) We propose a new task of learning generative models from a single video . 2 ) We present a novel two-branch discriminator , encouraging the generation of new scene compositions with layouts and content substantially different from training samples . Our proposed diversity regularization ensures a high variability among generated samples in the challenging single data instance regimes . 3 ) With SIV-GAN , we achieve high quality and diversity when learning from both single images and videos , outperforming prior work on two different datasets . 2 RELATED WORK . Single Image Generative Models . Several works have investigated learning generative models from a single image . Ulyanov et al . ( 2018 ) ; Shocher et al . ( 2018 ) showed that when trained on a single image , a deep convolutional network can learn a useful representation that captures the internal statistics of that image . These learned representations can be employed to synthesize textures from a sample texture image ( Jetchev et al. , 2016 ; Bergmann et al. , 2017 ; Ulyanov et al. , 2016 ; Zhou et al. , 2018 ; Li & Wand , 2016 ; Ulyanov et al. , 2017 ) , to ” blindly ” super-resolve ( Shocher et al. , 2018 ; Bell-Kligler et al. , 2019 ; Lugmayr et al. , 2020 ) , or to inpaint the image ( Ulyanov et al. , 2018 ) . Recently , single image GAN models ( Shocher et al. , 2019 ; Shaham et al. , 2019 ) have been proposed , revealing the power of image priors learned from a single natural image for synthesis and manipulation tasks . InGAN ( Shocher et al. , 2019 ) introduced a GAN model conditioned on a single natural image , which can remap the input to any size or shape while preserving its internal patch-based distribution . SinGAN ( Shaham et al. , 2019 ) employed an unconditional GAN to produce images of arbitrary size from noise , and used a multi-stage training scheme to learn the multi-scale patch distribution of an image . ConSinGAN ( Hinz et al. , 2021 ) extended SinGAN by improving the rescaling for multi-stage training and training several stages concurrently , which enabled reducing the size of the model and making the training more efficient . Alternatively , instead of using adversarial training Granot et al . ( 2021 ) proposed to use a patch-nearest-neighbour search . Contrary to single image GANs ( Shaham et al. , 2019 ; Hinz et al. , 2021 ) , our model is trained in a single stage , and is designed to learn not only the internal patch-based distribution of an image , but also to capture high-level content , such as scene layouts or appearance of objects . The latter enables more diverse , higher quality synthesis ( see Table 1 ) . Our model is also more universal : it achieves high quality and diversity when learning from both single images and videos , and thus can be applied to a wider range of real-world scenarios . Few-Shot Generative Models . Few-shot image generation seeks to generate more data of a given domain , with only a few training samples available . Many existing methods resorted to using large image sets of seen classes to train a conditional GAN , which is then used to generate new images for an unseen class based on a few examples ( Hong et al. , 2020a ; Antoniou et al. , 2017 ; Hong et al. , 2020b ) . Others focused on improving the sample efficiency of GANs by few-shot adaptation . These methods usually start from GANs pre-trained on large datasets , and adapt them on a few samples in the target domain by fine-tuning the generator and discriminator weights ( Wang et al. , 2018 ; Mo et al. , 2020 ) , changing batch statistics ( Noguchi & Harada , 2019 ) , restricting the space of trainable parameters ( Robb et al. , 2020 ) , transferring the knowledge via a miner network ( Wang et al. , 2020 ) , or enforcing cross-domain consistency ( Ojha et al. , 2021 ) . Unlike our work , the above methods heavily rely on external knowledge from a pre-trained GAN . Thus , their performance highly depends on the semantic consistency between the target domain and the pre-trained model , and their incompatibility can lead to even worse results ( Zhao et al. , 2020a ) . Most recently , FastGAN ( Liu et al. , 2021 ) proposed to train an unconditional GAN for few-shot image synthesis from scratch . To avoid overfitting , they enabled a fast learning of the generator through a skip-layer channel-wise excitation module , and used a self-supervised discriminator to continuously provide useful signals to the generator . However , FastGAN still suffers to successfully learn from a single image or video . Although the single video setting has similar number of training frames to the few-shot setting ( ∼100 samples ) , it provides much less variability in training data due to high correlation of adjacent frames , leading FastGAN to memorization issues ( see Fig . 5 ) . Another line of work focused on improving the stability of GANs when trained with limited data . To prevent the discriminator from overfitting , Karras et al . ( 2020a ) ; Zhao et al . ( 2020b ; a ) explored differentiable data augmentations ( DA ) for both real and generated images . Tseng et al . ( 2021 ) proposed to regularize GANs under limited data with a LeCam consistency loss , showing it to be complementary to DA techniques . These works used limited , but still relatively large training sets ( > 1k images ) compared to the few-shot and single video regimes ( 6 100 ) . As shown in Sec . 4.2 , using DA alone is not enough to achieve good diversity when learning from a single image or video .

Summary: This paper proposes an approach to train an image generator using only a short video clip or a single image. The overall framework follows the design of sinGAN that has an unconditional generator that generates new images from random noises, and the generated images and the input real images/frames are fed to discriminators to enforce similar distributions. The key contribution is to disentangle the image into content and layout features, and use different discriminators, which is claimed to reduce the memorization issue.