Data-Efficient Instance Generation from Instance Discrimination

1 Introduction . Generative Adversarial Network ( GAN ) [ 16 ] has become a popular paradigm to learn the distribution of the observed data . It is formulated as a two-player game , where a generator synthesizes realistic data , while a discriminator distinguishes synthesized samples from real ones . To reach equilibrium in this minimax game , it requires both the generator and the discriminator to be sufficiently trained . In other words , the synthesis capability of the generator will subsequently deteriorate given an inadequate discriminator [ 24 , 39 , 49 , 51 ] . Recent success of GANs [ 22 , 23 , 25 , 4 ] relies on big data to assure the sufficient training of the discriminator . Prior work [ 49 , 24 ] has found that reducing the amount of training data leads to the overfitting of the discriminator , which tends to memorize the entire training set . In turn , the backpropagation from the discriminator to the generator damages the synthesis quality of the generator and potentially causes the mode collapse problem [ 1 , 44 ] . Data augmentation is one of the most widely used methods to alleviate the overfitting issue in deep learning algorithms [ 45 , 11 , 10 ] . Some recent attempts [ 24 , 39 , 49 , 51 , 44 ] have been made to apply data augmentation to GAN training . It is found that the discriminator can be improved by augmenting not only the real images from the dataset but also the synthesized images by the generator [ 49 , 24 ] . However , the learning objective of the discriminator remains as categorizing real and fake domains and a substantial performance drop can be observed given limited training data . 1Code is available at https : //genforce.github.io/insgen/ . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The domain bi-classification task could be too easy for the discriminator to gain sufficient discriminative power as an adaptive loss to train the generator , especially when the size of training set is small . In this work , we propose to improve the data efficiency in GAN training by assigning a more challenging task to the discriminator , which is to distinguish every individual image as an independent category . In this way , the discriminator is forced to improve its discriminative capability to accomplish the instance discrimination task [ 40 ] . Notably , besides distinguishing real samples , we also demand the discriminator to differentiate fake samples synthesized by the generator . Thus the discriminator can be considered to train with infinite data , preventing it from memorizing the training samples . When distinguishing synthesized data , we design a noise perturbation strategy to increase the difficulty of the task and hence make the discriminator more capable . Meanwhile , we also alter the training objectives from the generator side . Concretely , besides making the generator to fool the discriminator , we expect all the samples produced by the generator to be well identified as different instances with our instance-induced discriminator . This highly matches the goal of diverse generation , which requires every synthesis to be unique . We evaluate our method on a range of datasets and achieve appealing generation performance in terms of image quality , diversity , and data efficiency . Experiments show that our method significantly improves the baselines and outperform previous data-augmentation methods . To be specific , our method improves the FID from 15.60 to 11.92 , 7.29 to 4.90 , and 3.88 to 3.31 with 2K , 10K , and 70K training images from FFHQ [ 23 ] respectively . We can even learn a large-scale GAN with only 100 in-the-wild images to produce satisfying synthesis . Our main contributions are summarized as follows : 1 ) We propose a data-efficient instance generation ( InsGen ) method which incorporates instance discrimination as an auxiliary task in GAN training . 2 ) The synthesized data is used as infinite samples for improving the discriminative power of the discriminator , which in turn substantially improves the synthesis quality and diversity of the generator . 3 ) Under various data-regime settings , our method consistently surpasses existing alternatives by a substantial margin . 2 Related Work . Data Augmentation in GANs . Data augmentation makes the maximum use of available data to alleviate the overfitting of deep models that have millions of parameters . It plays an essential role in training discriminative models [ 45 , 11 , 10 ] . Some recent work explores how data augmentation can help the training of GANs [ 51 , 39 , 49 , 24 ] . Zhao et al . [ 51 ] conduct empirical studies on the effects of different types of augmentations for GAN training . Tran et al . [ 39 ] make a theoretical analysis of several data augmentations . Zhao et al . [ 49 ] propose a differentiable augmentation method such that the augmenting operations can be applied to both real and synthesized data . Similarly , Karras et al . [ 24 ] design augmentations that do not leak and introduce a probability-based adaptive strategy to stabilize the training process . Different from prior work , we focus on introducing the unsupervised representation learning which also requires augmentations into GAN training . Our work shows that the recent instance discrimination task [ 40 ] can be used as an auxiliary task for the discriminator , which in turn substantially improves the synthesis quality of the generator . Self-supervised Learning in GANs . The rationale behind self-supervised learning is to set up various pretext tasks with supervisory-free labels [ 14 , 5 , 41 , 48 , 13 , 32 , 34 , 42 , 34 , 15 , 31 , 35 ] . Similar idea is recently introduced in GAN training as an auxiliary loss to improve the synthesis performance . For instance , Chen et al . [ 6 ] assign the rotation prediction task to the discriminator to prevent it from catastrophic forgetting , and Tran et al . [ 38 ] propose a multi-class minimax game to encourage the generator to produce diverse samples . Among all self-supervised learning approaches , contrastive learning [ 40 , 17 , 7 , 18 , 3 ] shows great potential in large-scale representation learning . Many attempts have been made to improve generative models by drawing lessons from contrastive learning , like the consistency regularization for GANs [ 47 , 50 ] , the patch-level contrastive learning for image-to-image translation [ 33 ] , and the latent-augmented contrastive loss for conditional image synthesis [ 29 ] . Akin to supervised contrastive loss [ 27 ] , some concurrent work [ 20 , 21 , 43 ] reformulates the conventional bi-classification task ( i.e. , real domain vs. fake domain ) with contrastive loss . Differently , we keep the original bi-classification task of the discriminator and introduce contrastive learning as a new one . Specifically , we assign the discriminator a simple auxiliary task , which is to recognize every individual image , no matter it is real or synthesized by the generator . Such instance discrimination task helps sustain the discriminative power of the discriminator under a low-data regime , which in turn improves the synthesis performance significantly . 3 Methodology . In this section , we introduce the proposed InsGen method . Recall that our method is built based on GAN , which is commonly formulated as a two-player game between a generator and a discriminator . They compete with each other in that the generator tries to produce as realistic data as possible while the discriminator works on recognizing synthesized data from real data . Besides the conventional bi-classification task ( i.e. , differentiating real and fake domains ) , we also require the discriminator to distinguish every individual instance . With such a challenging task , the discriminator can mitigate the overfitting problem even with limited training data . We will briefly introduce the image synthesis and instance discrimination mechanisms in Sec . 3.1 , followed by our improved training pipeline in Sec . 3.2 and the practical usage of InsGen on the state-of-the-art StyleGAN2-ADA model [ 24 ] in Sec . 3.3 . 3.1 Preliminaries . Our work is highly related to GAN [ 16 ] for image synthesis and contrastive learning [ 40 , 17 ] for instance discrimination . To make the paper self-contained , we shortly describe these two algorithms in the text below . Synthesizing Images with GANs . GAN is a popular paradigm for image generation . It typically consists of two networks : a generator G ( · ) that learns to map a latent variable z to a photo-realistic image , and a discriminator D ( · ) that aims at separating real images x from synthesized ones G ( z ) . These two networks compete with each other [ 16 ] and are jointly optimized with LD = −Ex∈X [ log ( D ( x ) ) ] − Ez∈Z [ log ( 1−D ( G ( z ) ) ) ] , ( 1 ) LG = −Ez∈Z [ log ( D ( G ( z ) ) ) ] , ( 2 ) where Z and X denote the pre-defined latent distribution and real data distribution respectively . After the training converges , the synthesized images are assumed to be as realistic as real ones to fool the discriminator . From this perspective , the synthesis quality highly depends on the discriminative power of the discriminator . Prior literature [ 24 , 39 , 49 , 51 ] has affirmed that GANs will suffer from the insufficient training of the discriminator and proposed to apply a series of data augmentations T ( · ) to alleviate the overfitting problem . But they do not change the learning objectives of GAN and observe drastic performance drop given limited training data . Distinguishing Images with Contrastive Learning . It is well-known that image classification tasks usually benefit from more discriminative representations [ 12 ] . Unlike supervised training algorithms that optimize the model parameters based on annotated data , contrastive learning [ 40 , 17 , 7 , 18 , 3 ] is able to extract representative features from images in an unsupervised manner . As shown in Fig . 1a , the rationale behind is to “ label ” every sample as an individual class , i.e. , instance discrimination . Concretely , given an image x , two random “ views ” ( e.g. , through different augmentations ) are created as the query xq and the key xk+ . This query-key pair is regarded as the positive pair while all “ views ” from other images , { xki } Ni=1 , are treated as negative pairs with respect to the query . Here , N is the total number of images in addition to the query image . Contrastive learning aims at maximizing the agreement across augmentations ( i.e. , xq and xk+ ) and make the query as much dissimilar to a number of negative samples as possible . Accordingly , we can design a pretext task of ( N + 1 ) -way classification and learn the model with the contrastive loss C i.e. , InfoNCE loss [ 32 ] vq = F ( xq ) , vk+ = F ( xk+ ) , vki = F ( xki ) , i = 1 . . . N , ( 3 ) CF ( · ) , φ ( · ) ( xq , xk+ , { xki } Ni=1 ) = − log exp ( φ ( vq ) Tφ ( vk+ ) /τ ) ∑N i=0 exp ( φ ( vq ) Tφ ( vki ) /τ ) , ( 4 ) where F ( · ) is the backbone network to extract the representation v from a given image x , and φ ( · ) is the head network ( e.g. , usually implemented with several fully-connected layers ) to project the extracted feature onto a unit sphere . τ stands for the temperature , which is a hyper-parameter . Recall that the primitive goal of the discriminator in GANs can also be viewed as a bi-classification task , which is to recognize real and fake domains . In this work , we demonstrate that introducing the instance discrimination task can help enhance the discriminative power of the discriminator and in turn improve the synthesis quality of the generator significantly .

Summary: This paper introduces a novel GAN training method, named InsGen. This work effectively overcomes the performance degradation problem of GAN training with a small amount of the train dataset. The main idea is to train a discriminator by assigning an instance discrimination task as well as a bi-classification task (e.g. real or fake). The experimental results show that, with the smaller amount of data used, the synthesized image quality of InsGen outperforms the images from SOTA methods.