Generating Novel Scene Compositions from Single Images and Videos

1 INTRODUCTION . The quality of synthetic images produced by generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) has greatly improved in recent years ( Zhang et al. , 2019 ; Brock et al. , 2019 ; Park et al. , 2019 ; Karras et al. , 2019 ; Lin et al. , 2019 ; Schönfeld et al. , 2020 ; Karras et al. , 2020b ) . These impressive results are in large part enabled by the availability of large , diverse datasets , typically consisting of tens of thousands of images . This dependency on the availability of training data limits the applicability of GANs in domains where collecting a large dataset is not feasible . In some realworld applications , collection of even a small dataset remains challenging due to specific constraints related to privacy , copyright status , subject type , geographical location , time , and dangerous or hazardous environments . It may happen that rare objects or events are present only in one image or video , and it is difficult to obtain a second one . For example , this includes images of exclusive artworks or videos of traffic accidents recorded in extreme conditions . Enabling GANs to learn in the extremely low data regimes , such as learning from the single data instance examples , thus has the potential to improve their utilization in practice . Prior work ( Shocher et al. , 2019 ; Shaham et al. , 2019 ; Hinz et al. , 2021 ) focused only on learning generative models from a single image ( Fig . 1 , rows 3-5 ) . In this work , we introduce a new task of learning to synthesize images from a single video ( Fig . 1 , rows 1-2 ) , where the training data is a collection of frames from a 2-10 second video clip . In practice , capturing such a short video takes almost as little effort as collecting a single image . However , compared to a single image , a video contains much more information about the scene and the objects of interest ( e.g. , different poses and locations of objects , various camera views , dynamic backgrounds ) . Learning from a video can enable synthesis of higher quality , more diverse images , while still keeping very low requirements on data availability , and therefore can improve GAN usability for practical applications . Training GANs in such extremely low data regimes is challenging as the models are prone to overfitting , leading to replication of training data or poor synthesis quality due to training instability ( Karras et al. , 2020a ) . For example , applying the few-shot image synthesis models , such as FastGAN ( Liu et al. , 2021 ) , to learn from a single image or video leads to severe memorization problems ( see Sec . 4 ) . To mitigate memorization , single-image GAN models ( Shaham et al. , 2019 ; Hinz et al. , 2021 ) proposed to learn an internal patch-based distribution of an image , employing a cascade of multiscale patch-GANs ( Isola et al. , 2017 ) trained in multiple stages . Though these models overcome memorization , producing different versions of a training image , they can not learn high-level semantic properties of the scene , e.g. , to dissect objects from the background . They often suffer from incoherent shuffling of image patches , distorting objects and producing unrealistic layouts ( see Fig . 4 and 5 ) . Furthermore , they have more difficulty to generate realistic scenes from a single video . We aim to go beyond patch-based learning , seeking to generate novel , plausible compositions of objects in the scene , while preserving the original image context . We want to learn a model which is able to compose new layouts , re-arrange objects in the scene , remove or duplicate instances , and change their shape and size . The new scene compositions should be visually plausible , with objects preserving their appearance and natural shape , and scene layouts looking realistic to the human eye . Reaching such synthesis quality from a single image or a single video is notably challenging , as the model needs to learn to distinguish objects and background using only one image or few training frames , without any direct object supervision ( e.g. , mask annotations ) . To this end , we introduce SIV-GAN ( Single Image and Video GAN ) , an unconditional , one-stage GAN model , capable of learning from the single data instances to generate images that are substantially different from the original training sample , while still preserving its context . This is achieved by two key ingredients : the novel discriminator design and the proposed diversity regularization for the generator . Our discriminator has two branches , separately judging image content and scene layout realism . The content branch evaluates objects ’ fidelity irrespective of their spatial arrangement , while the layout branch looks only at the global scene coherency . Disentangling the discriminator ’ s decision about content and layout helps to prevent overfitting and provides a more informative signal to the generator . To achieve a high diversity among generated samples , we further extend the regularization technique of ( Yang et al. , 2019 ) to unconditional image synthesis in single data instance regimes . The prior work of ( Yang et al. , 2019 ) encourages the generation of different images depending on their input latent codes , thus the difference between images is proportional to the distance between their codes in the latent space . Assuming that in case of a single image or a single video all generated images should belong to one semantic domain ( i.e . preserve the original training sample context ) , and thus should be more or less equally different from each other , we apply diversity regularization uniformly , independent of the latent space distance . Moreover , we use the regularization in the feature space , inducing both high- and low-level diversity . We demonstrate the effectiveness of our model for the single image setting as well as for the novel single video setting in Sec . 4 . SIV-GAN is the first model that successfully learns in both of these extremely low-data settings , improving over prior work ( Shaham et al. , 2019 ; Hinz et al. , 2021 ; Liu et al. , 2021 ) in both image quality and diversity . In contrast to FastGAN ( Liu et al. , 2021 ) , our model does not suffer from memorization ( see Table 1 and 2 ) ; while compared to single-image GANs ( Shaham et al. , 2019 ; Hinz et al. , 2021 ) , our model does not distort objects , preserving their appearance . In summary , our contributions are : 1 ) We propose a new task of learning generative models from a single video . 2 ) We present a novel two-branch discriminator , encouraging the generation of new scene compositions with layouts and content substantially different from training samples . Our proposed diversity regularization ensures a high variability among generated samples in the challenging single data instance regimes . 3 ) With SIV-GAN , we achieve high quality and diversity when learning from both single images and videos , outperforming prior work on two different datasets . 2 RELATED WORK . Single Image Generative Models . Several works have investigated learning generative models from a single image . Ulyanov et al . ( 2018 ) ; Shocher et al . ( 2018 ) showed that when trained on a single image , a deep convolutional network can learn a useful representation that captures the internal statistics of that image . These learned representations can be employed to synthesize textures from a sample texture image ( Jetchev et al. , 2016 ; Bergmann et al. , 2017 ; Ulyanov et al. , 2016 ; Zhou et al. , 2018 ; Li & Wand , 2016 ; Ulyanov et al. , 2017 ) , to ” blindly ” super-resolve ( Shocher et al. , 2018 ; Bell-Kligler et al. , 2019 ; Lugmayr et al. , 2020 ) , or to inpaint the image ( Ulyanov et al. , 2018 ) . Recently , single image GAN models ( Shocher et al. , 2019 ; Shaham et al. , 2019 ) have been proposed , revealing the power of image priors learned from a single natural image for synthesis and manipulation tasks . InGAN ( Shocher et al. , 2019 ) introduced a GAN model conditioned on a single natural image , which can remap the input to any size or shape while preserving its internal patch-based distribution . SinGAN ( Shaham et al. , 2019 ) employed an unconditional GAN to produce images of arbitrary size from noise , and used a multi-stage training scheme to learn the multi-scale patch distribution of an image . ConSinGAN ( Hinz et al. , 2021 ) extended SinGAN by improving the rescaling for multi-stage training and training several stages concurrently , which enabled reducing the size of the model and making the training more efficient . Alternatively , instead of using adversarial training Granot et al . ( 2021 ) proposed to use a patch-nearest-neighbour search . Contrary to single image GANs ( Shaham et al. , 2019 ; Hinz et al. , 2021 ) , our model is trained in a single stage , and is designed to learn not only the internal patch-based distribution of an image , but also to capture high-level content , such as scene layouts or appearance of objects . The latter enables more diverse , higher quality synthesis ( see Table 1 ) . Our model is also more universal : it achieves high quality and diversity when learning from both single images and videos , and thus can be applied to a wider range of real-world scenarios . Few-Shot Generative Models . Few-shot image generation seeks to generate more data of a given domain , with only a few training samples available . Many existing methods resorted to using large image sets of seen classes to train a conditional GAN , which is then used to generate new images for an unseen class based on a few examples ( Hong et al. , 2020a ; Antoniou et al. , 2017 ; Hong et al. , 2020b ) . Others focused on improving the sample efficiency of GANs by few-shot adaptation . These methods usually start from GANs pre-trained on large datasets , and adapt them on a few samples in the target domain by fine-tuning the generator and discriminator weights ( Wang et al. , 2018 ; Mo et al. , 2020 ) , changing batch statistics ( Noguchi & Harada , 2019 ) , restricting the space of trainable parameters ( Robb et al. , 2020 ) , transferring the knowledge via a miner network ( Wang et al. , 2020 ) , or enforcing cross-domain consistency ( Ojha et al. , 2021 ) . Unlike our work , the above methods heavily rely on external knowledge from a pre-trained GAN . Thus , their performance highly depends on the semantic consistency between the target domain and the pre-trained model , and their incompatibility can lead to even worse results ( Zhao et al. , 2020a ) . Most recently , FastGAN ( Liu et al. , 2021 ) proposed to train an unconditional GAN for few-shot image synthesis from scratch . To avoid overfitting , they enabled a fast learning of the generator through a skip-layer channel-wise excitation module , and used a self-supervised discriminator to continuously provide useful signals to the generator . However , FastGAN still suffers to successfully learn from a single image or video . Although the single video setting has similar number of training frames to the few-shot setting ( ∼100 samples ) , it provides much less variability in training data due to high correlation of adjacent frames , leading FastGAN to memorization issues ( see Fig . 5 ) . Another line of work focused on improving the stability of GANs when trained with limited data . To prevent the discriminator from overfitting , Karras et al . ( 2020a ) ; Zhao et al . ( 2020b ; a ) explored differentiable data augmentations ( DA ) for both real and generated images . Tseng et al . ( 2021 ) proposed to regularize GANs under limited data with a LeCam consistency loss , showing it to be complementary to DA techniques . These works used limited , but still relatively large training sets ( > 1k images ) compared to the few-shot and single video regimes ( 6 100 ) . As shown in Sec . 4.2 , using DA alone is not enough to achieve good diversity when learning from a single image or video .

Summary: The paper presents a new GAN model that generates new scene compositions from a single training image or a single video clip.  This is achieved by a two-branches (content-layout) discriminator and a diversity regularization technique. The paper also claims as the first work learning from a single video.