Data-Efficient Instance Generation from Instance Discrimination
Authors: Ceyuan Yang
Yujun Shen
Yinghao Xu
Bolei Zhou
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative adversarial network
data efficient training
image synthesis

Abstract: Generative Adversarial Networks (GANs) have significantly advanced image synthesis, however, the synthesis quality drops significantly given a limited amount of training data. To improve the data efficiency of GAN training, prior work typically employs data augmentation to mitigate the overfitting of the discriminator yet still learn the discriminator with a bi-classification ($\textit{i.e.}$, real $\textit{vs.}$ fake) task. In this work, we propose a data-efficient Instance Generation ($\textit{InsGen}$) method based on instance discrimination. Concretely, besides differentiating the real domain from the fake domain, the discriminator is required to distinguish every individual image, no matter it comes from the training set or from the generator. In this way, the discriminator can benefit from the infinite synthesized samples for training, alleviating the overfitting problem caused by insufficient training data. A noise perturbation strategy is further introduced to improve its discriminative power. Meanwhile, the learned instance discrimination capability from the discriminator is in turn exploited to encourage the generator for diverse generation. Extensive experiments demonstrate the effectiveness of our method on a variety of datasets and training settings. Noticeably, on the setting of $2K$ training images from the FFHQ dataset, we outperform the state-of-the-art approach with 23.5\% FID improvement.

Data-Efficient Instance Generation from Instance Discrimination
Ceyuan Yang!
Yujun Shen
Yinghao Xu Bolei Zhou
'The Chinese University of Hong Kong 'ByteDance Inc.
Abstract
Generative Adversarial Networks (GANs) have significantly advanced image synthesis, however, the synthesis quality drops sigmificantly given a limited amount of training data. To improve the data efficiency of GAN training, prior work typically employs data augmentation to mitigate the overfitting of the discriminator yet still learn the diseriminator with a bi-classification (i.e., real vs. fake) task. In this work, we propose data-efficient Instance Generation (InsGen) method based on instance discrimination. Concretely, besides differentiating the real domain from the fake domain, the discriminator is required to distinguish every individual image, no matter comes from the training set or from the generator. this way, the discriminator benefit from the infinite synthesized samples for training, alleviating the overfitting problem caused by insufficient training data. A noise perturbation strategy is further introduced to improve its discriminative power. Mcanwhile, the learned instance discrimination capability from the discriminator is in turn exploited to encourage the generator for diverse generation. Extensive experiments demonstrate the effectiveness of our method on a variety of datasets and training settings. Noticeably, the setting of 2K training images from the FFHQ dataset, we outperform the state-of-the-art approach with 23.5% FID improvement."
Introduction
Generative Adversarial Network (GAN) [16] has become a popular paradigm to learn the distribution of the observed data. It is formulated two-player game, where generator synthesizes realistic data, while a discriminator distinguishes synthesized samples from real ones. To reach equilibrium in this minimax game, it requires both the generator and the discriminator to be sufficiently trained. In other words, the synthesis capability of the generator will subsequently deteriorate given an inadequate discriminator [24, 39, 51].
Recent success of GANs [22, 23, 25, 4] relies on big data to assure the sufficient training of the discriminator. Prior work [49, 24] has found that reducing the amount of training data leads to the overfitting the discriminator, which tends to memorize the entire training set. turn, the back- propagation from the discriminator to the generator damages the synthesis quality of the generator and potentially causes the mode collapse problem [1, 44]. Data augmentation one of the most widely used methods to alleviate the overfitting issue in deep learning algorithms [45, 11, 10]. Some recent attempts [24, 39, 49, 51, 44] have been made to apply data augmentation to GAN training. is found that the discriminator dataset but also the synthesized images by the generator [49, 24]. However, the learning objective of the discriminator remains categorizing real and fake domains and substantial performance drop can be observed given limited training data.
be improved by augmenting not only the real images from the
Code available at https://genforet waw.loorrregggtitte github. io/insgen/.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The domain bi-classification task could be too easy for the discriminator to gain sufficient discriminative power as an adaptive loss to train the generator, especially when the size of training set is small. In this work, we propose to improve the data efficiency in GAN training by assigning a more challenging task to the discriminator, which is to distinguish every individual image as an independent category. In this way, the discriminator is forced to improve its discriminative capability to accomplish the instance discrimination task [40]. Notably, besides distinguishing real samples, we also demand the discriminator to differentiate fake samples synthesized by the generator. Thus the discriminator can be considered to train with infinite data, preventing it from memorizing the training samples. When distinguishing synthesized data, we design noise perturbation strategy to increase the difficulty of the task and hence make the discriminator more capable. Meanwhile, we also alter the training objectives from the generator side. Concretely, besides making the generator to fool the discriminator, we expect all the samples produced by the generator to be well identified as different instances with our indauce-induued discriminator: This highly matches the goal of diverse generation, which requires every synthesis to be unique. We evaluate our method on a range of datasets and achieve appealing generation performance in terms of image quality, diversity, and data efficiency. Experiments show that our method significantly improves the baselines and outperform previous data-augmentation methods. To be specific, our method improves the FID from 15.60 to 11.92, 7.29 to 4,90, and 3.88 to 3.31 with 2K, 10K, and 70K training images from FFHQ [23] respectively. We can even learn large-scale GAN with only 100 in-the-wild images to produce satisfying synthesis.
Our main contributions are summarized as follows: 1) We propose a data-efficient instance generation (InsGen) method which incorporates instance discrimination GAN training. 2) The synthesized data is used as infinite samples for improving the discriminative power of the discriminator, which in turn substantially improves the synthesis quality and diversity of the generator. 3) Under various data-regime settings, our method consistently surpasses existing alternatives by a substantial margin.
auxiliary task in
2 Related Work
Data Augmentation in GANs, Data augmentation makes the maximum use of available data to alleviate the overfitting of deep models that have millions of parameters. It plays an essential role in training discriminative models [45, 11, 10]. Some recent work explores how data augmentation can help the training of GANs [51, 39, 49, 24]. Zhao et al. [51] conduct empirical studies the effects of different types of augmentations for GAN training. Tran et al. [39] make a theoretical analysis of several data augmentations. Zhao et al. [49] propose a differentiable augmentation method such that the augmenting operations can be applied to both real and synthesized data. Similarly, Karras et al. adaptive strategy to
[24] design augmentations that do not leak and introduce a probability- stabilize the training process. Different from prior work, we focus on introducing the unsupervised representation learning which also requires augmentations into GAN training. Our work shows that the recent instance discrimination task [40] can be used as an auxiliary task for the discriminator, which in turn substantially improves the synthesis quality of the generator.
Self-supervised Learning in GANs. The rationale behind self-supervised learning is to set up various pretext tasks with supervisory-free labels [14, 5, 41,48, 13, 32, 34, 42, 34, 15, 31, 35). Similar idea is recently introduced in GAN training as an auxiliary loss to improve the synthesis performance. For instance, Chen et al. [6] assign the rotation prediction task to the discriminator prevent it from catastrophic forgetting, and Tran et al. [38] propose multi-class minimax game to encourage the generator to produce diverse samples. Among all self-supervised learning approaches, contrastive learning [40, 17, 7, 18, 3] shows great potential in large-scale representation learning. Many attempts have been made to improve generative models by drawing lessons from contrastive learning, like the consistency regularization for GANs [47, 50]. the patch-level contrastive learning for image-to-image translation [33], and the autent-augmened contrastive loss for conditional image synthesis [29]. Akin to supervised contrastive loss [27], some concurrent work [20, 21, 43] reformulates the conventional bi- ci-clasiffication task (i.e., real domain v.s. fake domain) with contrastive loss. Differently, we keep the original bi-classification task of the discriminator and introduce contrastive learning as new one. Specifically, we assign the discriminator a simple auxiliary task, which to recognize every individual image, no matter it is real or synthesized by the generator. Such instance discrinination task helps sustain the discriminative power of the discriminator under a low-data regime, which in turn improves the synthesis performance significantly.
2

---

(a) Triin Discriminator with Instance Diseriminaaiio
G(
G frozen
Instance Discrimination
G
Zi
D
Lo + CJ + Co.
(G,
(oy
(b) Train Generator with Instance Discrimination
((P,)
Z1
G(Zn
G
D
LG + C
Z
G
Dozen
Figure 1: Illustration of the InsGen method. Besides the bi-classification task to differentiate real and fake domains, the discriminator is assigned auxiliary task, which aims at maximally distinguishing each image instance as illustrated on the right. C denotes the training objective for such instance discrimination task. (a) The discriminator is asked to recognize not only every real sample x; but also every synthesized sample (iz;) by frozen generator. (b) With the instance-inducee discriminator, the generator is encouraged to make all synthesis recognizable from each other, leading to more diverse generation.
3 Methodology
In this section, we introduce the proposed InsGen method. Recall our method built based on GAN, which is commonly formulated as two-player game between generator and discriminator. They compete with each other in that the generator tries to produce as realistic data as possible while the discriminator works on recognizing synthesized data from real data. Besides the conventional bi-classification task (i.e., differentiating real and fake domains), we also require the discriminator to distinguish every individual instance. With such challenging task, the discriminator the overfitting problem even with limited training data. We will briefly introduce the image synthesis and instance discrimination mechanisms in Sec. 3.1, followed by improved training pipeline in the-art art StyleGAN2-ADA model [24] in
mitigate
Sec. 3.2 and the practical usage of InsGen on the state-of saate-of-the-ase
Sec. 3.3.
3.1 Preliminaries
Our work is highly related GAN [16] for image synthesis and contrastive learning [40, 17] for instance discrimination. To make the paper self-contained, we shortly describe these two algorithms in the text below.
Synthesizing Images with GANs. GAN is a popular paradigm for image generation. It typically consists of two networks: generator G(-) that learns to map a latent variable peoto-realistic image, and a discriminator D(.) that aims at separating real images x from synthesized ones G(z). These networks compete with each other [16] and are jointly optimized with
LD Exxxx log( D(x)) - Ezez log((1 - D(G(z)))]
(1)
LG Ezez E (((()))),
(2)
where Z and X denote the pre-defined latent distribution and real data distribution respectively. After the training converges, the synthesized images are assumed to be as realistic as real ones to fool the discriminator. From this perspective, the synthesis quality highly depends on the discriminative power of the discriminator. Prior literature [24, 39, 49, 51] has affirmed that GANs will suffer from the insufficient training of the discriminator and proposed apply series of data augmentations T(-) to alleviate the overfitting problem. But they do not change the learning objectives of GAN and observe drastic performance drop given limited training data.
Distinguishing Images with Contrastive Learning. well-known that image classification tasks usually benefit from more discriminative representations [12]. Unlike supervised training algorithms

---

that optimize the model parameters based on annotated data, contrastive learning [40, 17, 7, 18, (8.33 is able to extract representative features from images in an unsupervised manner. As shown in Fig. la, the rationale behind is to "label" every sample as an individual class, i.e., instance discrimination. Concretely, given an image x, two random "views" (e.g. through different augmentations) are created as the query X, and the key Xky- This query-key pair is regarded as the positive pair while all "views" from other images, {xx, } total number of images in addition to the query image. Contrastive learning aims at maximizing the agreement across augmentations (i.e.. X2 and xx+) and make the query much dissimilar to number of negative samples as possible. Accordingly, we can design pretext task of (N + 1) w)-way classification and learn the model with the contrastive loss C i.e. InfoNCE loss [32]
treated as negative pairs with respect to the query. Here, N is the
V4 F(Xy),
VR++
F(XX+),
Vk: =F(xx;); i= 1....,
(3)
Cp())(((Xx X (= = -lo log exp(o(v,) (v(Vk.)/7)
(4)
Σi= exp(o(v,) ('v(()/)))
where F(-) is the backbone network to extract the representation v from a given image x, and ((-) is the head network (e.g. usually implemented with several fully-connected layers) to project the extracted feature onto a unit sphere. T stands for the temperature, which is a hyper-parametce Recall that the primitive goal of the discriminator in GANs can also be viewed which is to recognize real and fake domains. In this work, we demonstrate that introducing the instance discrimination task can help enhance the discriminative power of the discriminator and in turn improve the synthesis quality of the generator significantly.
bi-classification task,
3.2 Generating Diverse Instances from Distinguishing Instances
In this part, we will introduce how instance discrimination is incorporated into the GAN training for data-efficient and diverse image generation. There are four essential components of our InsGen method: 1) distinguishing real images, 2) distinguishing fake images that can be sampled infinitely, 3) a noise perturbation strategy, and 4) a loop-back mechanism to encourage the generator for the diverse generation.
Distinguishing Real Images. As discussed above, the synthesis quality of GAN models not only depends on the training scheme [2, 30, 22, 4] and the architecture design of the generator [46, 23, 25]. but more importantly relies on the discriminative capability of the discriminator. That is because the discriminator is the only one (compared to the generator) that can see how real data looks like and further guides the generator accordingly. To make the maximum use of the limited training data and avoid the discriminator from memorizing the entire dataset, we assign it with a more challenging task beyond domain classification, which is to recognize every independent instance from the dataset, as shown in Fig. la. For this purpose, we introduce new task head ø (.) beyond the original bi- bi-classification head zdomain()) (-) on top of its backbone d(-)2 and train the discriminator with an extra training objective
CT = Ca(),or (Tg(x))) Te, (xy), (Ta, (xx,))
(5)
Here, X (Xx, Ji=1 are all sampled from the real data distribution X' and transformed with various differentiable augmentations (-)).
Distinguishing Fake Images. However, the amount of training data could be extremely few (like thousands or even hundreds) in practice. In such case, the improvement of the discriminator gained by differentiating real instances will be also limited. On the other hand, we notice that the number of synthesized samples can be sufficiently large due to the sampling mechanism of GANs. Ideally, different latent codes z Z should lead to different synthesis G(z). Hence, we propose to also ask the discriminator to recognize every individual fake images, as shown in Fig. la. Similarly, we introduce another task head di (-) into the discriminator. It is worth mentioning that we use separate task heads (i.e., o" (-) and of (-)) for real and fake data. That is because even though the synthesized images can be with high-quality, they still lie in different distribution from the real ones, especially when the generator starts training from scratch. Meanwhile, the task of discriminating a real instance from a fake instance can be achieved by the native domain classification head domaai (-).
The conventional discriminator is a composition of d(-) and @domain (-) to perform real/fake classification. Le.. D()) = ddomain (.) d(-).

---

Noise Perturbation. Prior work has observed the continuity of the latent space [36] such that images synthesized from the latent codes within a neighbourhood are very close to each other. Accordingly. they are more suitable to be treated as positive pairs than negative pairs. From this perspective, we introduce a noise perturbation strategy into fake image discrimination. The objective becomes
x (G())))) Xx, = TT- + +)), Xx. = TR, (G(zk)))
(6)
c/ = Cu().oor) (xx)))r))xxxx000) X,,, (xx,)i=1
(7)
Concretely, given a query image x the key image xx_ is created with Tk_ (G(z, + e)) instead of Tk_ (G(z,)). Here, stands for the perturbation term, which is sampled from a Gaussian distribution whose variance is sufficiently smaller than that of 2. and (-(+) and Tk_ (-) denote two different augmentations. Such design aims to enforce the discriminator invariant to the small perturbation, which makes the instance discrimination task more challenging.
Toward Diverse Generation. Besides utilizing the instance discrinination task to improve the discriminative power of the discriminator, we further design loop-back mechanism in turn the learned instance discrimination to guide the generator. Recall that image diversity, in addition to image quality, is also an important metric to evaluate generative models. Diverse generation, which requires all generated samples to be distinguishable from each other, exactly matches our goal of instance discrimination. In other words, given a discriminator with the ability to distinguish different instances, we would like all the samples produced by the generator to be recognized as different ones. This idea is illustrated in Fig. lb. By comparing Fig. la and Fig. 1b, we can see that the generator shares the same target as the discriminator yet is trained separately. Hence, the same objective function added into the generator loss
X = Tki (G(z,)),
(8)
Cac (x,= XL_ (x)
(9)
where the only difference is that noise perturbation is not applied during the training of the generator. Complete Objective Function. To summarize, with the purposes of both image synthesis and instance discrimination, the discriminator and the generator in InsGen are optimized with
C LD XDCs X/cb-
(10)
c/ = La + DoC'
(11)
where Xa, Xo. and Xb denote the weights for different terms.
3.3 Implementation
On top of the adversarial training pipeline in GANs, InsGen method only inserts an extra loss output on the discriminator network for instance discrimination.Theraee implemented any GAN framework. In this part, we take the state-of-the-art GAN model, StyleGAN2-ADA [24], as an example to demonstrate how InsGen is implemented in practice.
it can be easily
Generative Model. StyleGAN2-ADA [24] adopts the architecture of StyleGAN2 [25] and proposes the adaptive discriminator augmentation strategy for training with limited data. In particular, designs a differentiable augmentation pipeline, consisting of 18 transformations, as well as an adaptive hyper-parameter to control the strength of these augmentations. For a fair comparison, in this work, we exactly reuse the network structure, the augmentation pipeline, the adaptive strategy of the augmenting strength, and other hyper-parameten like batch size and learning rate.
Instance Discrimination. We reuse the backbone of the discriminator to perform instance diserimination, so that the extra computing load is extremely small and the training efficiency is barely affected. We treat the last fully-connected layer in the StyleGAN2-ADD the domain- classification head ddoaai (.), while all remaining layers serve as the backbone network d(.). The real instance discrimination head o² (-) and the fake head & (-) are both implemented with 2 fully-connected layers, followed by &s normalization. Strictly following MoCo-v2 [8], an extra queue is employed for each task head store the sample features to save computational cost. The number of samples in LD and C/D is thus equal to the queue size, which usually contains around 5% data of the whole set. We also introduce the momentum encoder D', whose parameters are updated with moving average scheme: On +- a@p rr + (1 a)Qp. Here, a = 0.999 follows the same setting in MoCo-v2 v2 [8]. The temperature in Eq. (4) is set as 2.
discriminator as
5

---

Table 1: Performance on FFHQ. FID (lower is better) is reported as the evaluation metric. "2K", "10K", and "140K"" stand for the number samples used for training, where "140K" horizontally flips the original FFHQ dataset (with 70K samples) to double the size of data. Results with are also achieved with horizontally flipped data, which are slightly better than those reported [24]. Numbers in blue color indicate our improvements over the baseline [24].
256x256 Resolution
2K
10K
140K
PA-GAN [44]
56.49
27.71
3.78
zCR [50]
71.61
23.02
3,45
Auxiliary rotation [6]
66.64
25.37
4.16
StyleGAN2 [23]
78.80
30.73
3.66
w/ Shallow mapping [24]
71.35
27.71
3.59
w/ Adaptive dropout [24]
67.23
23.33
4.16
w/ DiffAugment [49]
24.32
7.86
w/ ADA [24]
15.60*
7.29*
3.88
InsGen (Ours)
11.92 (-3.68)
4.90 (-33))
3.31 (-0.57)
4 Experiments
We evaluate the proposed InsGen method on multiple benchmarks. Sec. 4.1 presents the comparison to prior literature on both FFHQ [23] and AFHQ [9] datasets. Our InsGen substantially improves the baselines under multiple data-regime settings and outperforms previous tata-augmentation approaches by a significant margin. Moreover, Sec. 4.2 provides a detailed ablation study to show the importance of each component. Lastly Sec, 4.3 discusses about the limitation of data-efficiency.
4.1 Main Results
Datasets. We evaluate our InsGen with number of other approaches on FFHQ [23] and AFHQ [9] datasets. FFHQ contains unique 70,000 high-resolution images (1024x 1024), with large variation regarding age, ethnicity, and background. All images of FFHQ are well aligned [26] and cropped. In order to conduct fair comparison,
resize images to x56x250 For the experiments of limited data, we follow ADA [24] to collect a subset of training data by randomly sampling. Morcover, AFHQ consists of around 5000 images per category for dogs, cats, and wild life 512x 512 resolution. Each category regarded as a dataset and thus we train a different network on each dataset.
Training. We implement our InsGen on the official implementation of StyleGAN2-ADA. The training regularization is preserved, including path length regularization, lazy regularization, and style mixing regularization. Moreover, all parameters share the same learning rate and the minibatch standard deviation layer adopted at the end the discriminator. Exponential moving average of generator weights, non-saturating logistic loss with RS regularization, and Adam optimizer [28] is also adopted. In particular, the coefficient of gradient penalty would be decreased correspondingly, according to the official implementation of ADA [24]. All the experiments are conducted on server with GPUs Mixed-precisio training is also used for faster training.
Hyper-parameters. Empirically, the loss weights ÀG. X and XT are 0.1, 1.0 and 1.0 respectively. Besides, the training length is slightly different. For the experiments with less than 10K images, the total number of seen images is 10 million rather than 25 million adopted by ADA [24]. Mcanwhile, we decrease the loss weight of the gradient penalty due to involving an extra supervision. For example, ADA [24] adopts 1.0 for original StyleGAN2 training while we use 0.8. We also found smaller loss weight of gradient penalty is beneficial our InsGen on the less data, e.g., 0.3 and 0.5 for 10K and 2K experiments respectively.
Evaluation Metric. We use Fréchet Inception Distance (FID) [19] the metric for quantitative comparison metric since FID tends to reflect the human perception of synthesis quality. As mentioned in Heusel et al. [19], we always calculate the FID between 50,000 fake images and all training images, no matter how much data the training set contains. The official pre-trained Inception network is used to compute the FID.

---

Table 2:
Performance on AFHQ. FID (lower is better) is reported as the evaluation metric. Numbers blue color indicate our improvements over the baseline [24].
512x512 Resolution
Cat
Dog
Wild life
StyleGAN2 [23]
5.13
19.4
3.48
ContraD [20]
3.82
7.16
2.54
ADA [24]
3.55
7.40
3.05
InsGen (Ours)
2.60
(-0.95)
5.44 (- 1.96)
1.77 (- (-1.28)
140K. FID 3.31
10K, FID 4,90
2K. FID 11.92
FFHQ-256
Cat (5153), FID 2.60
Dog (4739), FID 5.44
Wild life (4738). FID 1.77
AFHQ AFHQ-312 312
Figure Generated images under various data regimes. The number of training images and the corresponding FID are reported. All images on FFHQ are synthesized with truncation following [24] while those on AFHQ are not.
Results FFHQ. Tab. presents the comparison FFHQ. Akin to ADA [24), compare against PA-GAN [44], zCR [50] and auxiliary rotation [6]. Also, StyleGAN2 together with its variants is also introduced as the baseline methods. For instance, less data is usually required when shallower mapping network is applied. Besides, dropout [37] also well-studied to be replaced with the augmentations the regularization. Note that means the dataset is amplified by 2x via the horizontal flip, which is recommended in the official implementation of ADA [24). Such that, "2K"" denotes 2,000 unique images and the dataset is enlarged to 4,000 via the flip operation, leading to better bascline.
Although ADA [24] has already improved the performance significantly under various low-data regimes, our InsGen continues to improve the low-data image generation by a clear margin, establishing
state-of-ther the-art synthesis quality with limited training images. To be specific, method improves the FID from 15.60 to 11.92, 7.29 to 4.90, and 3.88 to 3.31 with 2K, 10K and 70K training images from FFHQ [23] respectively. Fig. 2 presents several generated examples under various data regimes. More qualitative results are available in our supplementary material. All images on FFHQ are generated with truncation. It also worth noting that approach further improves the synthesis quality when the full dataset is given, even outperforming previous hest one i.e., zCR [50]. Namely, the data can be further exploited when it is not the bottleneck for training.
Results on AFHQ. We also evaluate our approach on AFHQ dataset [9] which divided into cat, dog and wild life, with the number of 5153, 4739 and 4738 images respectively. Therefore, three models are trained on them individually. Note that all models on AFHQ are trained on 512x512 images while the generated samples resized present. We involve StyleGAN2 [25], ContraD [20] and ADA [24] as the baseline approaches, compared to our InsGen. Quantitative and qualitative results shown in Tab. 2 and Fig. 2 respectively.
The synthesis quality on those datasets substantially improved by our method, which also outperforms previous data auta-tuumeetttiion methods. To be specific, our method improves the FID

---

Table 3: Ablation Study. FID (lower is better) is reported as the evaluation metric. Here, vanilla C/2 means that the noise perturbation is not applied in the fake instance discrimination.
cD
vanilla Co
C/
C22
2K
10K'
70K'
15.60
7.29
3.76
14.15
5.98
3.56
13.46
5.68
3.67
12.19
5.30
3.49
/
11.92
4.90
3.31
from 3.55 to 2.60, 7.40 5.44, and 3.05 1.77 cat, dog and wild life images respectively. In particular, ContraD [20] introduced stronger augmentations to train a better discriminator via contrastive learning. One term in this method shares the similar motivation that real images could result in powerful representations. In terms of the use of synthesized samples, ContraD turned to focus on the binary classification, (i.e., real vs. fake) with some specific designs like the stop-gradient operation. Differently, our method leverages the generated images as kind of data complement to produce a stronger representation and guide the learning of the generator. Accordingly, InsGen achieves the new state-of- satte-of-the-ree tht-hee-rtt art performances on AFHQ [9].
4.2 Ablation Study
In order to investigate the importance of each component in our InsGen, we conduct an ablation study on FFHQ [23] with the image resolution of 256x256. FID serves as the main metric for the comparison, and the results on 2K, 10K and 70K unique images are reported. During training each unique image go through random flip operation to obtain stronger baseline. Tab. 3 presents the collection of various experiments the ablation study. We choose the ADA [24] as the baseline.
How important is the instance discrimination? After performing the real image discrimination, the synthesis quality is improved, with the FID consistently decreased by -1.45, -1.31 and -0.20 Tab. no matter how many unique images the training set includes. To some extent, the discriminator would benefit from the powerful representations derived from the challenging pretext task. Accordingly, the generator is required to produce more photo-realistic images in order to confuse the discriminator.
When adding instance discrimination with fake images, performances could be further boosted. For instance, FID obtains an improvement of -0.69 and -0.30 with 2K and 10K images respectively. In particular, the gains rise the number of real images goes down, verifying of motivations that the fake samples can be also regarded as data source for unsupervised representation learning.
How important is the noise perturbation? In Sec. 3.2, noise perturbation strategy is proposed as a type of latent space augmentation for fake image discrimination. In particular, this latent space augmentation, i.e., the small movement in the latent space always leads an obvious but semantically consistent change of the original image, which could not easily be implemented by some geometric and color transformations. Meanwhile, the discriminator is required be invariant such noise perturbation due to the goal of instance discrimination. Accordingly, the fake images are made best of to result in stronger representations for the discrimination. As shown in Tab. such strategy further brings consistent gains -1.27, -0.38 and -0.18 on 2K, 10K and 70K datasets respectively.
How important is the supervision signal for the generator? The last row of Tab. 3 shows the performances with the gradients which are back poopppropagtted to the generator. Even if we have already obtained quite strong results, such supervision signal on the generator could also introduce improvements under various data regimes.
The goal of instance discrimination is to distinguish every individual image according to its appearance cues [40]. Assuming this pretext task is perf-performed on fixed dataset, the semantic representation would be derived from this learning process. However, when distinguishing fake images, the fake dataset actually varies dynamically. Namely, we could accomplish this pretext task from perspective of data, if the engine of this dynamical fake dataset, i.e., the generator could produce as many different images as possible. In general, this pretext task exploited encourage the diverse generation directly on the generator.
8

---

13:45
$7.51
29.50
$6,49: PA-GAN
30.73: StyleGAN2
13.07
19.58
15,60: ADA
FID
FTD
12.49
11.52
11.92
4.90
3.31
400
000
800
1000
100
250
500
1X
70K
(a) #Synthesized Samples for Discrimination
(b) #Real Samples Discrimination
Figure 3: Effect of the number of synthesized and real images used for instance discrimination. FID (lower is better) in log-scale is reported as the evaluation metric. We can see the consistent performance gain along with the increasing number of instances for discrimination.
4
Eo
Ous--Real
AlDA-Real
ADA-Fake
Clurs- Fulke
Iteaations
Figure Training progress on FFHQ-2K. Larger value means that the image more realistic under the view of the discriminator, Our discriminator can better and more stably differentiate real and fake data compared ADA [24].
How important the number of negative samples? We follow the MoCo-v2 [8] to store multiple features in queue, in order to reduce the computational complexity. Empirically, the length of the feature queue tends to 5% number of the dataset. Therefore, it 200 when we have 2K unique images and enlarge them via the flip operation. However, there is no any reference number for the synthesized data. Accordingly, we collect as same amount of fake data as that the real.
As mentioned in Sec. 3.2, there could be much more synthesized samples than the real samples. Namely, we could leverage infinite samples for the synthesized instance discrimination. Therefore, we investigate the effect of the different number of synthesized samples i.e., the length of the feature queue, shown in Fig. 3a. Obviously, FID gradually decreases with the increasing number of synthesized samples, suggesting that involving more fake images is of great benefit to the synthesis, especially with the limited training data.
Whether the discriminative ability of the discriminator is really enhanced. As is mentioned in work, it is challenging to gain sufficient discriminative power for the diseriminator train the generator when the size of training set is small. However, introducing instance discrimination is able improve its discriminative capability, achieving new state-of-the -hee---fthh-rttt synthesis quality. In order to investigate whether the discriminative ability is improved, we plot the logits (derived from the discriminator) of any input image during the training in Fig. To be specific, the logit denotes how much the input image is identified the real. And the number of training images are 2000.
Obviously, method produces higher real and lower fake scores throughout the whole training progress, compared to the baseline approach ADA [24]. It indicates that the discriminator of our method performs the domain bi-classification (i.e., real fake) better than that of baseline, showing stronger discriminative ability. also verifies our motivation that a challenging pretext task which is distinguish every individual image could indeed enhance the discriminator. Besides, the training progress is much more stable when equipped with our approach.
4.3 Towards the Limit of Data-efficiency
Although we have obtained the new state-of-the-art synthesis performances under the standard settings, also wonder how much data-cfficiency our InsGen could achieve. Therefore, the number

---

1000. FID 19.58
500, FID 29.50
250, FID 37.51
100, FID 53.93
Figure 5: Qualitative results with different number of training images. The number of training images and the corresponding FID are reported. All images are synthesized with truncation following [24].
of real data in the training set is further reduced to 1000, 500, 250 and 100. In order to conduct the apple-to-apple comparison, we remain train the model of StyleGAN2 without decreasing its generative capacity by using fewer channels or shallower mapping networks since such designs require less data. Meanwhile, the generated resolution remains 256x250 and the datasets amplified via the horizontal flip operation as well.
The quantitative and qualitative results are shown in Fig. 3b and Fig. 5 respectively, Obviously, FID significantly increases with the decreasing number training images from 70K to 100. Nevertheless, InsGen trained with only 100 unique images remains outperform many approaches like PA-GAN in Fig. 3b with 2K images. Besides, with 500 training samples, our method is able to obtain the competitive performance those using 10k images. Namely, InsGen could improve the data-efficiency by more than 20x. Qualitative results suggest that our approach still produces meaningful images without incurring the model collapse matter how many training images exist in the data collection.
Conclusion and Discussion
In this work, we develop a novel data-efficient Instance Generation (InsGen) method for training GANs with limited data. With instance discrimination as an auxiliary task, our method makes best use of both real and fake images to train the discriminator. In turn the discriminator is exploited to train the generator to synthesize as many diverse images as possible. Experiments under different data regimes show that InsGen brings substantial improvement over the baseline in terms of both image quality and image diversity, and outperforms previous data augmentation algorithms by large margin.
Although InsGin significantly improves the data efficiency in training generative models, leaves some future work to do. One limitation of InsGen is that the performance gain becomes marginal when the training dataset sufficiently large. This suggests that the discriminator can not benefit from the newly introduced instance discrimination any more. It may require more challenging task to further improve the performance. Another limitation is that the FID score remains unsatisfying when the training data is extremely limited, say several hundred. It is worth exploring how to fully utilize the fake samples for discriminator training.
Acknowledgments. The project was supported through the Research Grants Council (RGC) of Hong Kong under ECS Grant No.24206219, GRF Grant No. 14204521, CUHK FoE RSES Grant.
References
[1] M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks. Int. Conf. Learn. Represent., 2017.
10

---

[2] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In Int. Conf. Much. Learn., 2017. 4
[3] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. Adv. Neural Inform. Process. Syst., 2019. 2.4
[4] A. Brock, Donahue, and Simonyan. Large scale gan training for high fidelity natural image synthesis. In Int. Conf. Learn. Represent, 2018. 1,
[5] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, P. Dhariwal, D. Luan, and Sutskever. Generative pretraining from pixels. In Int. Conf. Mach. Learn., 2020. 2
[6] T. Chen, X. Zhai, M. Ritter, M. Lucie, and N. Houlsby, Self-supervised gans via auxiliary rotation loss. In IEEE Conf. Comput. Vis. Pottern Recog., 2019, 2, 6,
[7] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In lnt. Conf. Mach. Learn., 2020. 2, 4
X. Chen, H. Fan, Girshick, and K. Hc. Improved baselines with momentum contrastive leamingg arXiv preprint or orXiv:2003.044 044997 2020. 5,9
[9] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. Stargan v2: Diverse image synthesis for multiple domains, In IEEE Conf. Comput. Vis. Pattern Recog., 2020. 6, 7,8
[10] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Lc. Autoaugment: Learning augmentation policies from data. In IEEE Conf. Comput. Vis. Pattern Recog., 2018. 1.2
[11] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with reduced search space. In IEEE Conf. Comput. Vis. Puttern Recog. Worksh., 2020. 1
[12] J. Deng, W. Dong, R. Socher. L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. JEEE Conf. Comput. Vis. Pattern Recog., 2009. 3
[13] C. Doersch, Gupta, and A. Efros. Unsupervised visual representation learning by context prediction. In Int. Conf. Comput. Vis., 2015. 2
[14] J. Donahue and K. Simonyan. Large scale adversarial representation learning. In Adv. Neural Inform. Process. Syst., 2019. 2
[15] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In Int. Conf. Learn. Represent. 2018. 2
[16] T. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. In Adv. Neural Inform. Process. Syst., 2014. 1,3
[17] K. Hc, Fan, Wu, S. Xic, and Girshick. Momentum contrast for unsupervised visual representation
learning. IEEE Conf. Comput. Vis. Pattern Recog., 2020. 2, 3,
[18] O. Henaff. Data-efficient image recognition with contrastive predictive cnding. In Int. Conf. Mach. Learn., 2020. 2,4
[19] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Adv. Neural Inform. Process. Syst., 2017. 6
[20] J. Jeong and J. Shin. Training gans with stronger augmentations via contrastive discriminator. In Int. Conf. Learn. Represent., 2021. 2. 7,8
[21] M. Kang and J. Park. Contragan: Contrastive learning for conditional image generation. In Adv. Neural Inform. Process. Syst. 2020.
[22] T. Karras, T. Aila, Laine, and Lehtinen. Progressive growing gans for improved quality, stability, and variation. In Int. Conf. Learn. Represent., 2018. 1,4
[23] T. Karras, S. Laine, and T. Aila. A styl-based generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019. 1, 4, 6, 7,8
[24] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. In Ady Neural Inform. Process. Syst., 2020. 2, 5, 6. 8, 9, 10
11

---

[25] T. Karras, S. Laine. M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In IEEE Conf. Comput. Vis. Pattern Recog, 2020. 1, 4, 7,
[26] V. Kazemi and J. Sullivan. One millisecond face alignment with an ensemble of regression trees. In IEEE Conf. Comput. Vis. Pattern Recog.. 2014. 6
[27] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. In Adv. Neural Inform. Process. Syst., 2020.
[28] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Int. Conf. Learn. Represent., 2014. 6
[29] R. Liu, Y. Ge, C. L. Choi, X. Wang, and H. Li. Divco: Diverse conditional image synthesis via contrastive generative adversarial network. In IEEE Conf. Comput. Vis. Pattern Recog. 2021. 2
[30] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In Int. Conf. Learn. Represent., 2018. 4
[31] M. Naroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In
Eur. Conf. Comput. Vis., 2016. 2
[32] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: (tt::8007.0330 03748, 2018. 2,4
[33] T. Park, A. A. Efros, R. Zhang. and J.-Y. Zhu. Contrastive learning for unpaired image-to-image translation. In Eur, Conf. Comput. Vis., 2020. 2
[34] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In IEEE Conf. Comput. Vis. Pattern Recog., 2016. 2
[35] D. Pathak, R. Girshick, P. Dollár, T. Darrell, and B. Hariharan. Learning features by watching objects move. In IEEE Conf. Comput. Vis. Pattern Recog., 2017. 2
[36] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Int. Conf. Learn. Represent., 2016. 5
[37] N. Srivastava, G. Hinton, A. Krizhevsky, Sutskever, and Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 2014." 7
[38] N.-T. Tran, V.H.. Tran, N.-. Nguyen, L. Yang, and N.MM Cheung. Self-supervieed gan: Analysis and improvement with multi-class minimax game. In Ady. Neural Inform. Process. Syst., 2019. 2
[39] N.-T. Tran, V.H.. Tran, N.-B. Nguyen, T.-K. Nguyen, and N.-M. Cheung, On data augmentation for gan training. IEEE Trans. Image Prucess., 2021. 1,2,3
[40] Z. Wu, Y. Xiong, X. Yu, and Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE Conf. Comput. Vis. Pattern Recog., 2018. 2, 3,4,8
[41] Y. Xu, Y. Shen, Zhu, Yang, and B. Zhou, Generative hierarchical features from synthesizing images. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[42] C. Yang, Z. Wu, B. Zhou, and S. Lin. Instance localization for self-supervised detection pretraining. In IEEE Conf. Comput. Vis. Pattern Recog., 2021, 2
[43] N. Yu, G. Liu, A. Dundar, A. Tao, B. Catanzaro, L. Davis, and M. Fritz. Dual contrastive loss and attention for gans, arXiv preprint arXiv:2 103. 16748, 2021, 2
[44] D. Zhang and A. Khoreva. Pa-gan: Improving gan training by progressive augmentation. In Adv. Neural Inform. Process. Syst. 2019. 6,7
[45] H. Zhang, M. Cisse, N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In Int. Conf. Learn. Represent., 2017. 1.2
[46] H. Zhang, 1. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial networks. In Int. Conf. Mach. Learn 2019,
[47] H. Zhang, Z. Zhang, A. Odena, and H. Lec. Consistency regularization for generative adversarial networks. In Int. Conf. Learn Represent., 2020. 2
[48] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In Eur. Conf. Comput. Vis., 2016. 2
12

---

[49] S. Zhao, Z. Liu. J. Lin, J.-Y. Zhu, and S. Han. Differentiable augmentation for data-efficient gan training.
In Adv. Neural Inform. Process. Syst., 2020. 2, 6
[50] Z. Zhao, S. Singh, H. Lee, 7. Zhang. A. Odena, and H. Zhang. Improved consistency regularization for gans. In Assoc. Adv. Artif. Intell., 2020. 2, 6.
[51] Z. Zhao, Z. Zhang, T. Chen, S. Singh, and H. Zhang. Image augmentations for gan training. arXiv preprint arXiv:2006. 02595, 2020. 1, 3
13


Reviews:
[{"review_id": "y-ycDJzS5sl", "reviewer": "Reviewer_JJnj", "summary": "The paper proposes a kind of contrastive loss that aims to discriminate both real and generated instances for GAN training. Experiments on FFHQ and AFHQ demonstrate that the proposed method implemented upon StyleGAN2-ADA can improve FID especially with limited data.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed method sounds novel and well motivated.\n- The proposed method can improve StyleGAN2-ADA's FID especially with limited data. It is good to see that the proposed method can also improve its FID on the full FFHQ training set.\n\nWeaknesses:\n- My main concern is that the authors did not provide any empirical evidence on the generator diversity as the authors claimed. With limited data, the generator can memorize the full training set very easily, i.e., the generator may simply produce identical training images and thus achieve a low FID. Thus it is necessary to conduct nearest neighbor tests, style space interpolation, or measure the diversity quantitatively using e.g. precision/recall.\n- I think the proposed loss combination is a bit complex but not fully justified. For example, why can't we simply apply data augmentation to fake images as done in real images instead of noise perturbation?\n- The experiments are limited to the StyleGAN2-ADA backbone, so it is not clear whether the proposed method can generalize over other GAN architectures/algorithms. The datasets are also a bit limited.\n- In the supplementary material, only the fact that the discriminator achieves better accuracy on the training set is not meaningful.\n\n====== Post Rebuttal ======\n\nChanged to 6 provided that the authors have promised to include diversity experiements in the revision and addressed my remaining concerns.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Good", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "VwhwjHXjqIK", "reviewer": "Reviewer_9ouC", "summary": "This paper propose to add instance discrimination (aka contrastive learning) objective for training generative adversarial networks. Specifically, (a) instance discrimination among real samples & (b) instance discrimination among fake samples are used as two axillary objectives. In addition, noise perturbation whereby the method uses z & z+epsilon to create two views needed for contrastive learning. (conventional contrastive learning uses 2 different augmentations of an image to create 2 views). Experimental results show that the proposed method InsGen outperforms prior methods that use data augmentations for training image GANs.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n- Overall, the paper is well-written and easy to follow.\n- Using noise perturbation to create two views is pretty novel.\n- The method can have high impact to the field, as it is simple but highly effective. I think it is much easier to use compared to ContraD --- which is another method that used contrastive learning for GAN training. Personally, I have tried using ContraD for my project, but it was not easy to implement. This method can be a good alternative to ContraD.\n\n[Weaknesses]\n- Novelty and Comparison against prior art: There is prior works in GAN literature that used instance discrimination objectives: ContraD [20] and DC-VAE [a]. Conceptually, InsGen is pretty similar to these works.\nI believe comparison against these works should be written explicit in the paper.\nThis submission (InsGen) has subtle differences against ContraD:\n(1) ContraD does not use instance discrimination for training the generator. InsGen does.\n(2) A novel way of creating two views (noise perturbation).\n(3) ContraD uses SimCLR (for real samples) & SupCon (for fake samples), whereas InsGen uses MoCo.\n\nI think (1) & (2) are addressed in ablation, but (3) is not well-studied. The effect of different contrastive learning methods should be investigated. It is possible that the performance difference between ContraD and InsGen is due to the difference between SimCLR and MoCo.\n\nAlso, Table 1 (FFHQ) does not include ContraD. ContraD should be compared.\n\n- Missing details about augmentation. What kinds and strengths of data augmentation operations are used for MoCo? Does it follow the original setting of MoCoV2?\n\n- Missing results on CIFAR10. Since CIFAR10 is the most widely studied benchmark for Image GANs, it is better to include it.\n\n[Missing References]\n[a] Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In CVPR, 2021.\n\n======= Post-Rebuttal ======\nAfter reading the latest results, I increased my rating from 5 to 6 for following reasons:\n(a) the improvement is due to the difference in formulation (not the underlying contrastive learning method);\n(b) this method is cleaner and easier to use than ContraD.\nThe formulation of ContraD is not natural and kind of hard to implement (from my experience). Although ContraD is a similar work on introducing contrastive learning to the GAN training, I believe the community will benefit from easier-to-use and better performing variant. Hence, I vote for accepting this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations or potential negative societal impact are not included in this paper. Including limitations as well as the position of this paper against prior literature will greatly enhance the quality of this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "828uQhZTTlV", "reviewer": "Reviewer_HMNP", "summary": "This paper introduces a novel GAN training method, named InsGen. This work effectively overcomes the performance degradation problem of GAN training with a small amount of the train dataset. The main idea is to train a discriminator by assigning an instance discrimination task as well as a bi-classification task (e.g. real or fake). The experimental results show that, with the smaller amount of data used, the synthesized image quality of InsGen outperforms the images from SOTA methods.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n* This paper is well written and easy to follow up.\n* The idea of noise perturbation is novel (and clever!). \n* The idea of enhancing discriminator by the instance discrimination does help enhance overall generation quality of GANs. \n* Experimental results show a significant improvement when training GANs using a small amount of data. \n\nWeakness\n* Adding the experiments using datasets with various classes would be more convincing. At least, the experiment on cifar-10 should be appended. \n* This method does not seem to require a specific model structure, isn't it? It seems that the training method of InsGen can be applied not only to StyleGAN v2 but also to other structures such as BigGAN. However, in the current information, it is not clear. (If the method does not require specific model architectures, the authors should advertise themselves.)\n* According to Tab.3, the 70K FID of the 3rd-row increases (which means quality degradation) compared to the 2nd-row. Any explanation?  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "7w6Efu_X_5a", "reviewer": "Reviewer_eSj9", "summary": "This work adds an auxiliary contrastive learning task to train GANs in the regime of limited data. \nIn particular, they add an instance discrimination task to the discriminator and update both, generator and discriminator weights based on the modified objective. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe paper integrates well-known ideas from contrastive learning into GAN training to improve performance, particularly in the low-data regime. While the idea is simple, combining both fields is a very interesting approach. The related work is well-written and gives the reviewer a good overview of the current state of the field and this work's contribution.\n\nQuality: \nOverall, the claims made in this paper are well supported, e.g. by the ablation in Section 4.2. However, the experimental evaluation could be more thorough: \n   - To support the generalizability of this approach it would be good to extend the evaluation to a second architecture, e.g. https://arxiv.org/abs/2101.04775.\n   - Please discuss the following limitations: \n     i ) how does the contrastive objective impact the training time? Since the loss is computed over various samples does this slow down training or increase memory significantly? \n     ii) could this additional objective prevent the GAN from converging to the optimal solution? How important is the weighting of the losses and how are the hyperparameters from the appendix selected?\n    - Which choices of augmentations are most effective? Are the same augmentations needed as for ADA or are some less effective for the task of instance discrimination?\n   - Additional ablations that should be discussed: \n      i) If more synthetic samples keep improving FID (Figure 3), why stop at 1k? If feasible please add more results or discuss why this is not feasible in practice.\n     ii) What is the value of epsilon (missing in hyper-parameters in the appendix) and how sensitive is the approach to this choice? Can you include some samples within the range of epsilon to give the reader an intuition on how similar these samples are?\n      iii) Did you try using the same head for real/fake images (phi^r, phi^f)? While the given intuition (L.163) is comprehensible, an ablation study in the appendix would be good.\n      iv) Why is the noise perturbation not applied to the generator? You could either give an intuition or add results of this ablation to the Table 3.\n     v) Why do you use a momentum encoder? How much does this alone affect the results? Is this the standard (e.g. does ADA also use it?) or sth that you propose to do in this work? \n\nClarity:\nThe paper is well-written and well-structured. In particular, I like that it is self-contained and that the related work gives a good overview of the field.\nMinor: L.207, L_D^r and L_D^f are not defined previously\n\nSignificance: \nThe results from this work might be useful to the community because it proposes a relatively simple method to improve performance in the low-data regime. To judge the practical value of the proposed method, it would be good to include an analysis on the overhead wrt training time / memory consumption regarding the added auxiliary task.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned above, please discuss the limitations of your work, in particular, wrt training time and memory consumption. Also please discuss the societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {"ethical_concerns": "Ethical concerns are not discussed in this work. However, there are important issues that should be discussed at least shortly. In particular because this work improves image synthesis *in the low-data regime*."}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper should be accepted, given it's effectiveness. I agree with the reviewers. The contents of the rebuttal are important to distinguish the paper from previous work (especially ContraD) and therefore should be incorporated into the final version.
Decision: Accept (Poster)