Collapse by Conditioning: Training Class-conditional GANs with Limited Data
Authors: Mohamad Shahbazi
Martin Danelljan
Danda Pani Paudel
Luc Van Gool
Venue: ICLR 2022 Poster (2022)
Keywords: Generative Adversarial Network
GAN
Conditional GAN
limited data

Abstract: Class-conditioning offers a direct means to control a Generative Adversarial Network (GAN) based on a discrete input variable. While necessary in many applications, the additional information provided by the class labels could even be expected to benefit the training of the GAN itself. On the contrary, we observe that class-conditioning causes mode collapse in limited data settings, where unconditional learning leads to satisfactory generative ability. Motivated by this observation, we propose a training strategy for class-conditional GANs (cGANs) that effectively prevents the observed mode-collapse by leveraging unconditional learning. Our training strategy starts with an unconditional GAN and gradually injects the class conditioning into the generator and the objective function. The proposed method for training cGANs with limited data results not only in stable training but also in generating high-quality images, thanks to the early-stage exploitation of the shared information across classes. We analyze the observed mode collapse problem in comprehensive experiments on four datasets. Our approach demonstrates outstanding results compared with state-of-the-art methods and established baselines. The code is available at https://github.com/mshahbazi72/transitional-cGAN

Published as a conference paper at ICLR 2022
COLLAPSE BY CONDITIONING: TRAINING CLASS- CONDITIONAL GANS WITH LIMITED DATA
Mohamad Shahbazi, Martin Danelljan, Danda Pani Paudel & Lue Van Gool Computer Vision Lab (CVL), ETH Zurich, Switzerland (mshahbazi, wwwtaii.ii... danell jan, paudel, vangool } @vi sion, ee ethz. ftchz.ch ch
ABSTRACT
Class-condiiioning offers a direct means to control a Generative Adversarial Net- work (GAN) based on a discrete input variable. While necessary in many ap- plications, the additional information provided by the class labels could even be expected to benefit the training of the GAN itself. On the contrary. we observe that cladsicondiiioning causes mode collapse in limited data settings, where unconditional learning leads satisfactory generative ability. Motivated by this observation, we propose a training strategy for class-conditional GANs (cGANs) that effectively prevents the observed mode-collapse by leveraging un- conditional learning. Our training strategy starts with an unconditional GAN and gradually injects the class conditioning into the generator and the objective function. The proposed method for training cGANs with limited data results not only in stable training but also in generating high-quality images, thanks the early-stage exploitation of the shared information across classes. We ana- lyze the observed mode collapse problem in comprehensive experiments on four datasets. Our approach demonstrates outstanding results compared with state- of-the-art methods and established baselines. The code is available at httpp:
INTRODUCTION
Since the introduction of generative adversarial networks (GANs) by Goodfellow et al. (2014). there has been substantial progress in realistic image and video generation. The contents of such gen- eration are often controlled by conditioning the process by means of conditional GANs (Mirza & Osindero 2014). In practice, conditional GANs are of high interest, as they can generate and control wide variety of outputs using single model. Some example applications of conditional GANs include class-conditioned generation (Brock et al.) 2018), image manipulation Yuu et al. 2018), image-to-ing image translation (Zhu et al. 2017). and text-to-image generation (Xu et al. 2018
Despite the remarkable success, training conditional GANs requires large training data, including conditioning labels, for realistic generation and stable training (Tseng et al. (021). Collecting large enough data is challenging in many frequent scenarios, due to the privacy, the quality, and the diversity required, among other reasons. This difficulty often worsened further for datasets for conditional training, where also labels need to be collected. The case of fine-grained conditioning adds an additional challenge for data collection, since the availability of the data samples and their variability are expected to deteriorate with the increasingly fine- ine-grained details (Gupta et al.
2019
While training GANs with limited data has recently received some attention (Karras al. 2020a Wang et al. 2018 Tseng et al.) (202I), the influence of conditioning in this setting remains unex- plored. Compared to the unconditional case, the conditional information provides additional super- vision and input to the generator. Intuitively, this additional information can guide the generation process better and ensure the success of conditional GANs whenever its unconditional counterpart succeeds. In fact, one may even argue that the additional supervision by conditioning can even alleviate the problem of limited data, to an extent. Surprisingly, however, we observe an opposite effect in our experiments for clas-cconiiionall GANs. As visualized in Fig. [[ the conditionl GAN trained on limited data suffers from severe mode collapse. Its unconditional counterpart, on the other hand, trained the same data, is able generate diverse images of high fidelity with

---

Published as a conference paper at ICLR 2022
ImageNet Carnivores
Food101
CUB-200-2011
AnimalFaee Face
FID
2000
400
200
Unconditional
Conditional
Ours
Unconditional
Conditional
Class Cluss
Ours
Figure I: FID curves (first row) and sample images for training StyleGAN2+ADA unconditionally (second row), conditionally (third row), and using method (fourth row) four datasets un- der the limited-data setup (from left to right: ImageNet Carnivores, Food101, CUB-200-2011, and AnimalFace). The vertical axis of FID plots is in log scale for better visualization.
stable training process. To our knowledge, these counter-intuitive observations of class-conditional GANs have not been observed or reported in previous works.
In this paper, we first study the behavior of state-of-the-art class-conditional GAN, with varying the number of classes and image samples per class, and contrast it the unconditional case. Our study in the limited data regime reveals that the unconditional GANs compare favorably with conditional ones, in terms of the generation quality. We, however, are interested the conditional case, so as to be able to control the image generation process using single generative model. In this work, we, therefore, set out to mitigate aforementioned mode collapse problem.
Motivated by our empirical observations, we propose method for training class-conditional GANs that leverages the stable training of the unconditional GANs. During the training process, we inte- grate gradual transition from unconditional to conditional generative learning. The early stage of the proposed training method favors the unconditional objective for the sake stability, whereas the later stage favors the conditional objective for the desired control over the output by conditioning. Our transitional training procedure only requires minimal changes in the architecture the existing ofte-ooftthe-rt GAN model.
2

---

Published as a conference paper at ICLR 2022
We demonstrate the advantage of the proposed method over the existing ones, by evaluating method on four benchmark datasets under the limited data setup. The major contributions this study are summarized as follows:
We identify and characterize the problem of conditioning conditioning-indueed induced mode collapse when training class-conditional GANs under limited data setups.
We propose training method for class-conditional GANs that exploits the training stability of unconditional training to mitigate the observed conditioning collapse.
The effectiveness the proposed method is demonstrated on four benchmark datasets. The method is shown significantly outperform the oottc-of-thh-rrt and the compared baselines.
Conditional
Unconditional
Conditinal
Unconctional
Curs
11
122
133
70
24
1.2
106
Classes
Samplie
(a)
(b)
Figure 2: The FID scores for different experiments on ImageNet Carnivores using unconditional, conditional, and our proposed training of StyleGAN2+ADA by varying (a) the number of classes (number of samples per class is fixed at 100) and (b) the number of images per class (the number of classes is fixed 50). The total number of images for the experiments shown on the data points. The horizontal axis in Fig. (b) is in log scale for better visualization.
2 CLASS-CONDITIONINS MODE COLLAPSE
Training conditional image generation networks is becoming an increasingly important task. The ability to control the generator fundamental feature many applications. However, even in the context of unconditional GANs, previous studies suggest that class information as extra supervi- sion can be used to improve the generated image quality (Salimans et 2016 Zhou et al. 2018 Kavalerov et al.) 2020). This, in turn, may set an expectation that the extra supervision by con- ditioning must not lead to the mode collapse of cGANs in setups where the unconditional GANs succeed. Furthermore, one may also expect to resolve the issue of training cGANs on limited data, to an extent, due to the availability of the additional conditional labels.
As the first part of our study, we investigate the effect of class conditioning on GANs under the limited data setup. We base our experiments on StyleGAN2 with adaptive data augmentation (ADA), which is recent state-of-the- art method for unconditional and class caas-coonitional image generation under limited-data setup (Karras et al., 2020a]] Both unconditional and conditional versions of StyleGAN2 trained four benchmark datasets (more details in Section 4.1), with the setup of this paper. The selected datasets are somewhat fine-grained, where the problem of limited data, concerning their availability and labeling difficulty, is often expected to be encountered.
In Fig. T we analyze the training of conditional and unconditional GANs by plotting the Fréchet inception distance (FID) (Heusel al. (017)) during training. The analysis is performed on four dif- ferent datasets, each containing between 1170 and 2410 images for training. Contrary to our initial expectation, the conditional version consistently yields worse FID compared the unconditional one during the training. To analyze the cause, we visualize samples from the best model attained
'We originally also considered BigGAN (Brock al. 2018) another haseline. However, we found it to struggle with limited data in both unconditional and conditional settings.
3

---

Published as a conference paper at ICLR 2022
Style-mapping Network
Class Embeddings
2
Linear
L
Class Embeddings
PC
G
D
L
Linear
uC
Figure 3: The proposed modified training objective and architecture of StyleGAN2 allows for tran- sitioning from the unconditional to the conditional model during the training.
during training, in terms of FID, in Fig. [ For each dataset, the unconditional model learns to generate diverse and realistic images, while lacking the ability to perform class-conditional genera- tion. On the other hand, the conditional model suffers from severe mode collapse. Specifically, the intra-class variations are very small and mainly limited to color changes, while retaining the same structure and pose. Moreover, the images lack realism and contain pronounced artifacts.
Next, we further characterize the mode collapse problem observed in Fig. T by analyzing its de- pendence the size of the training dataset. To this end, employ the ImageNet Carnivores dataset (Liu et al. 2019)), which includes a larger number of classes and images. We perform the analysis by gradually reducing the size of the training set in two ways. In Fig. [p. we reduce the number of classes while having 100 training images in each class. In Fig. 2b. we reduce the number of images per class while using 50 classes all cases. both cases, the conditional GAN achieves a better FID for larger datasets, here above 5k images. This observation is in line with previous work (Salimans al. 2016 Zhou et al. 2018 Kavalerov et al. 2020). However, when reducing the data size, the order is inverted. Instead, the unconditional model achieves consistently better FID, while the conditional model degrades rapidly.
Inspired by these observations, we set out to design a training strategy for class-conditional GANs that eliminates the mode collapse induced by the conditioning. As visualized Fig. our proposed approach, presented next, achieves stable training, leading to low FID. The images generated from our model exhibit natural intra-class variations with substantial changes in pose, appearance, and color. Furthermore, training strategy outperforms both the conditional and unconditional models in terms of FID a wide range of dataset sizes, as shown in Fig. 2
METHOD
3.1 FROM UNCONDITIONAL TO CONDITIONAL GANS
As the analysis Section 2 reveals, class-conditional GAN training leads mode collapse when training data is limited, while the unconditional counterpart achieves good performance for the same number training samples. This discovery motivates us to design an approach capable of leverag- ing the advantages of unconditional training in order to prevent mode-collapse in conditional GANs. Our initial inspections of the generated images during conditional training indicate that the mode collapse appears from the very early stages of the training. On the contrary, the corresponding unconditional GAN learns to generate diverse images with gradually improved photo-realism during training. In order to avoid mode-collapse, of the training process. Unconditional training in the early stages allows the GAN model to learn the distribution of the real images without the complications caused conditioning, We then introduce the conditioning in the later stages of the training. This allows the network to adapt to conditional generation in stable way by exploiting the partially learned data distribution.
aim to exploit unconditional learning at the beginning
4

---

Published as a conference paper at ICLR 2022
In general, we control the transition from unconditional training to conditional training using transition function A 2 0. The subscript t denotes the iteration number during training. Specifically, A = 0 implics purely unconditional learning, i.c. the conditional information does not affect the generator or discriminator networks. Our goal is to design a training strategy capable of gradually incorporating conditioning during training by increasing the value of A₁ While any monotonically increasing function A may be chosen, we only consider a simple linear transition from 0 to 1 as,
6 [-TT T T.-7::00 :7
A = min max
Here, T, and Te respectively denote the time steps, at which the transition starts and ends. More details on the proposed transition function are provided in the Appendix, Sec. A. We achieve the desired transition during training by introducing method for controlling the behavior of the generator and the discriminator using the function AІ. An overview our approach, detailed in the next sections, is illustrated in Fig.
3.2 TRANSITION IN THE GENERATOR
The generator G(z) of a GAN is trained to map the latent vector 2, drawn from a prior distribution p(z), to generated image I G(2) belonging to the distribution of the training data. A condi- tional generator G(z,, c) additionally receives the conditioning variable as input, aiming to learn the image distribution of the training data conditioned on c.
The transition from an unconditional G(=) to a conditional G(2, c) generator seemingly requires discrete architectural change during the training process. We circumvent this by additionally con- ditioning our generator on the transition function A as G(z, e, Ae). More specifically, we gradually incorporate conditional information during training by using a generator of the form,
G(2,c, c, Ao) G(S(2) + X E(c)).
Here, S and are neural network modules that transform the latent and condition vectors, respec- tively. In case of A 0, the conditional information is masked out, leading to purely uncondi- tional generator (=)() During the transition step T. < t < Te, the importance of the conditional information gradually increased during training.
To achieve the generator in equation 2 we perform a minimal modification to the original class- conditional StyleGAN2 (Karras al. 2020b), where the generator consists of style-mapping net- work and an image synthesis network. The stylc-mapping network maps the input latent vector and the embedding of the condition e to the intermediate representation w, known as style code. The image synthesis network then generates images from the style codes. As illustrated in Fig. 3 we modify the conditioning in the generator by feeding the class embeddings to a fully-connected layer, which is then weighted by X, before adding to the output of the style-mapping network's s first layer.
3.3 TRANSITION IN THE TRAINING OBJECTIVE
The unconditional and conditional GAN frameworks also differ in their training objectives. The latter's objective assesses the realism of an image based its conditioning variable. We propose using both unconditional and conditional objectives and follow the same transition as in the genera- Our proposed loss function includes the unconditional objective during the whole training. The conditional term, on the other hand, is weighted by the transition function A Our total objective is thus given by.
L - L + L2_ _= LG + X, LG
Here, Le L., and D are the unconditional, conditional, and proposed losses for the discriminator, respectively. Similarly, IGG. IG, and G represent the unconditional, conditional, and proposed losses for the generator.
As the discriminator is responsible to predict the scores needed for calculating the loss, the proposed training objective requires also modifying the discriminator's 's architecture. In this regard, the nec- essary modification must provide both conditional and unconditional scores. We propose using two
5

---

Published as a conference paper at ICLR 2022
prediction branches, separately dedicated conditional and unconditional cases, in the last layer of the discriminator, as shown in Fig. 3
The proposed training objective and discriminator for StyleGAN2 are also visualized in Fig. In the discriminator of StyleGAN2, conditioning is performed by matching the features of the imput images with the target class embedding. In other words, the conditional discriminator assigns scores to the input images by calculating the dot-product between the features of the input images and the embedding of the target class c. The proposed discriminator architecture bears some resemblance to the architecture of projection discriminators (Miyato & Koyama (2018)). In contrast to our ap- proach, the projection discriminator aggregates the unconditional and conditional scores inside the discriminator before the loss function. Additionally, the projection discriminator does not perform any transition between the two scores.
EXPERIMENTS
In this section, we first provide the details of our experimental setup. Then, present the quantita- tive and qualitative results of the proposed method, as well as the comparison with existing methods. Finally, we provide more ablation and analysis of different components of our method.
4.1 EXPERIMENT SETUP
Datasets: We use four datasets to evaluate our method: ImageNet Carnivores (Liu et al. 2019). CUB-200-2011 (Wah et al.) 2011), Food101 (Bossard et al.] (014), and AnimalFace (Si & Zhu, 2011]. To keep our experiments in the limited-data regime, we decrease the number of classes and images per class in some of these datasets using random sampling.
ImageNet Carnivores is a subset of the ImageNet dataset (Deng et al. 2009), which con- tains 149 classes of carnivore animals. The images are further processed to only contain the animal faces. We use a subset of the dataset with 20 classes and 100 images per class. Food101 contains 101 different food categories, having total amount of 101k images. We use a subset of the dataset with 20 classes and 100 images per class.
CUB-200-201/ contains 200 categories of different bird species with around 60 images per class. We use a subset of this dataset containing 20 classes with all the images in each class. AnimalFace, similar ImageNet Carnivores, faces. However, the animals in AnimalFace are not limited to carnivores. AnimalFace contains 20 classes with 2432 images in total. Since AnimalFace is already small dataset, we do not further reduce its size.
dataset that contains images of animal
We base our method on the official PyTorch implementation of Style- GAN2+ADA. The hyper-parameter selection for the base unconditional and conditional StyleGAN2 performed automatically as provided in the official implementation. Training is done with a batch size of 64 using 4 GPUs. For the transition function, we use T, 2k and T. 4k in all experiments.
Implementation details:
Evaluation Metrics:
We evaluate our method using Fréchet inception distance (FID), as the most commonly-usee metric for measuring the similarity between the distribution of real and generated images. As FID can be biased when real data is small (Karras et al. 2020a), we also include kernel inception distance (KID) (Binkowski et al. 2018) metric that unbiased by design.
4.2 RESULTS AND COMPARISONS
To assess the efficacy of the proposed method, we provide a quantitative comparison with the well- established baselines and existing methods:
BigGAN+ADA (Brock et al. 2018): Achieving outstanding results on ImageNet, BigGAN has been widely used for class-conditional image generation (more details in Section 5
We use the implementation with ADA provided by (Kang & Park 2020).
ContraGAN++AA (Kang & Park 2020): A class-conditional model based on BigGAN that outperforms BigGAN in many setups using self-supervision in the discriminator (more
details in Section 5). We use the implementation provided by the authors.

---

Published as a conference paper at ICLR 2022
U-StyleGAN++AAA: Unconditional training of StyleGAA2++ADA using the original uncon- ditional architecture.
C-StyleGAN+ADAL Conditional training of StylcGAN2+AA using the original condi- tional architecture.
C-StyleGAN+ADAAppree The modified version of conditional StyleGAN2+ADA by
placing its discriminator with the projection discriminator (Miyato & Koyama 2018). C-SnieGAN++ADA-Lccmmm Regularizing C-StyleGAN2+ADA using Lecam regularizer, recently proposed by T'seng et al. (2021) for limited-datt setup (more details in Section [ The authors have suggested observe noticeable difference between different values in the suggested range, we set the hyper-parameter to 0.3 for all experiments.
hyper-paramae in the range of 0.5). As we did not
Results are reported in T BigGAN and ContraGANN even though coupled with ADA, struggle to achieve any good generation quality in our experiments. Conditional StyleGAN2 shows better results three of the datasets compared to that of the other two conditional competitors. However, the FID and KID scores are still very high. As discussed before, C-StyleGAN is consistently outper- formed by its unconditional counterpart. Replacing StyleGAN2's discriminator with the projection discriminator does not yield a noticeable advantage over the original architecture, as it brings im- provements two of the datasets, but degrades the performance the other two. Adding Lecam regularization to the C-StyleGAN2 shows promising results, achieving good FID and KID scores on Food101 and AnimalFace. However, still fails achieve as good generation quality as the unconditional StyleGAN2. The FID and KID scores for our proposed method indicate significant and consistent advantage over all the compared methods in all four datasets. Our method is able to maintain stable training and achieve better generation quality than both unconditional and con- ditional StyleGAN2. The FID curves during training along with the generated examples using the
Table I: Comparison of the proposed method with baselines and existing methods on four datasets in terms of FID and KID) metrics.
Method
Carnivores
Food101
CUB-200-2011
AnimalFace
FID
KID
FID
KID
FID
KID
FID
KID
BigGAN+ADA
97
0.0665
111
0.0794
136
0.0860
90
0.0587
ContraGAN+ADA
97
0.0629
124
0.0961
137
0.0934
89
0.645
UC-StyleGAN2++AA
23
0.0093
0.0071
27
0.0059
0.0048
C-SIYE:GAN2+ADA
100
0.0493
42
0.0135
55
0.0197
61
0.0107
C-SyyeGGAN2+ADA++Poojj
103
0.0503
32
0.0108
54
0.0182
71
0.0186
C-SYYaGAAN2-ADAAH..ccmm
62
0.0211
0.0086
0.0179
0.0042
Ours
14
0.0021
20
0.0045
22
0.0032
16
0.0018
proposed method are visualized in Fig. [ FID curves indicate training dynamics as stable as the un- conditional training while achieving better FID, In addition, the generated images of our method are clearly of more diversity and quality compared to those of the standard conditional model, showing the advantage of the proposed method. The results in Fig. 2 further demonstrate clear advantage of our method over a wide range of data sizes. Our approach maintains the performance of cGANs for larger datasets, while significantly outperforming the unconditional counterpart when data is more scarce. This shows that our method enables eGANs to use the additional label information to achieve better generation quality without falling into the mode collapse induced by conditioning.
4.3 ABLATION AND ANALYSIS
In this section, we provide further ablation and analysis over different components of our method. First, we provide an ablation study containing four different variants:
No transition: Training the modified architecture with the new objective, without any tran- sition in the objective or the generator (equivalent to using an auxiliary unconditional loss term to train conditional model).
Transition only in G: Performing the transition only in the generator (Sec. 3.2.. while the training objective is the summation of the unconditional and conditional term.

---

Published as a conference paper at ICLR 2022
Transition only in loss: Performing the transition only in the training objective (Sec. 3.3 while the generator is fully conditional from the beginning.
Final method: The final method with all the proposed components.
Table presents the results of the ablation study on Food101 and ImageNet Carivvores. The No transition version yields poor results, showing that the mode collapse is not alleviated by only adding an auxiliary unconditional training ob- jective. Adding the transition to the generator already brings significant improvement to the model. Having the transition only in the objec- tive, on the other, does not lead to good results. To our initial surprise, this reveals that transi- tioning in the generator is a crucial part of the method. However, transitioning in the objective in addition to that in the generator, as proposed in our final method, achieves the best results.
Table 2: Ablation study over different components of the proposed method, including the proposed architecture and objectives, as well as the transi- tion in the generator and in the objective.
Food101
ImageNet Carniv,
Experiment
FID
KID
FID
KID
No transition
79
0.0300
110
0.0436
Transition only in G Transition only in loss
25
0.0064
17
0.0019
80
0.0297
107
0.0539
Final method
0.0045
0.0021
Next, we analyze the impact of when the tran- sition between unconditional and conditional learning is applied. The total transition time is fixed to T. - T, = 2k time steps. We then re- port the results on the Food101 and ImageNet Carnivores datasets for different starting times T, in Table 3 Importantly, we notice signifi- cantly worse results if the transition is started at the beginning of the training T, = O. This further supports the hypothesis that condition- ing leads to mode collapse in the early stages of the training. By introducing conditional in- formation in a later stage, good FID and KID numbers are obtained without being sensitive to the specific choice of T. 2. In Table we further independently analyze the transition end time Te, while keeping T. = 2k fixed. Again, our approach is not sensitive to its value. Our method, therefore, does not require extensive hyper-parameter tuning.
Table 3: Analysing the importance of the transi- tion starting time (T.). The transition period constant at 2k for all the experiments.
Food101
ImageNet Carnivores
Experiment
FID
KID
FID
KID
T. 0
24
0.0068
27
0.0075
T. = 1k
22
0.0057
15
0.0023
T. = 2A:
20
0.0045
14
0.0021
T. = 4At
21
0.0056
14
0.0021
T. 64
23
0.0056
15
0.0028
Lastly, we visualize the evolution of the gener- ated images and the formation of classes dur- ing our training process. Fig 4 shows how im- ages generated from the unconditional phase of training on AnimalFace start to evolve into dif- ferent images of the class Panda. In addition to the formation of the classes, Fig. 4 shows how the image quality continues to improve during and after the transition. In the Appendix, more ablation studies (Sec. [A.6] (..)). as well as im- ages generated with our method (Sec. A.8) are provided for further assessment of the proposed method.
Table 4: Analysing the importance of the transi- tion ending time (Te). The starting time (T.) is constant at 2k for all the experiments.
Food101
ImageNet Carnivores
Experiment
FID
KID)
FID
KID
T. 3k
0.0053
16
0.0032
T. 4k
0.0045
0.0021
T. 5k
0.0062
15
0.0029
5 RELATED WORK
The first conditional GAN architecture, introduced by Mirza & Osin- dero (2014), incorporated conditioning by concatenating the condition variable to the input of the generator and the discriminator. AC-GAN (Odena et al., 2017) equipped the discriminator with an auxiliary classification task to ensure the conditional generation. cGAN with projection dis- criminator (Miyato& & Koyama, 2018) proposed a new discriminator architecture, ensuring the class conditioning by computing the dot-product between image features and class embeddings. Improv- ing over cGAN with projection discriminator, BigGAN (Brock et al.) 2018) was able to become the state-of-the-an eGAN on ImageNet dataset (Deng et al. 2009). Inspired by the recent progress in self-supervised learning, ContraGAN (Kang & Park 2020) improved over BigGAN by exploit-
Class-conditional GANs:
8

---

Published as a conference paper at ICLR 2022
1- 2016
2419
3024
4032
Figure 4: Visualization of the formation of the class "Panda" in AnimalFace during the transition from unconditional to conditional training. The transition starts at = 2k.
ing an auxiliary self-supervised task in the discriminator for better image representation learning. StyleGAN (Karras al., 2019), mainly known for unconditional image generation, was extended to class conditioning in the improved version, and coupled with adaptive differentiable augmentation (ADA), outperformed the state-of-the-ar art eGANs in small data setup (Karras et al., (202a)).
eGANs in small data regimes: There exist several directions to address the problem of training GANs on small data. Transfer learning (TL. exploits large pre-training data to provide better ini- tialization for learning the target data. Several works recently have studied the best practices for
TL in GANs (Wang et al.] 2018 2020; Mo et al., 2020 Zhao et al., 2020aa; Noguchi & Harada 2019). As example, cGANTransfer (Shahbazi et al. 2021) proposed class-specific knowledge
transfer in class-conditional GANs by learning the target class embeddings as linear combination
of the ones in the pre-trained model. Although effective, TL usually requires large pre-training data with sufficient domain relevance to the target. Data augmentation (DA) is another technique for addressing small data. To prevent DA from leaking to the generated images, recent works pro- posed differentiable DA (Zhao et al.] 20206 Karras et al., 2020a). Adding adaptive differentiable augmentation (ADA) to StyleGAN2 resulted in a significant improvement conditional generation from small datasets, outperforming previous models on CIFARIO (Karras et al. (2020a). In addition to the aforementioned methods, there are other studies focusing on better architecture or objective design for small data regimes. Liu et al. (2021) proposed a lighter unconditional architecture and a self-supervised discriminator for StyleGAN, seng al. (202T) proposed the Lecam regulariza- tion to prevent the discriminator from over-fitting on small data by penalizing the current difference between real and fake predictions from previous fake and real predictions, respectively.
Conditioning collapse in GANs: Previous studies have reported decrease in diversity in tasks with strong pixel-level conditioning. such as semantic masks or images (e.g. super-resolution) (Ramas- inghe et al. 2021: Lugmayr al. 2022 Lee et al. 2019 Isola et al. 2017). Such lack of diversity is generally considered to be due to the conflict between the adversarial and reconstruction losses common in maae--onnditonn. GANs. The same effect, however, not only is not explored for the class-conditional setting but also does not directly translate to this setup. In this work, we discover the conditioning collapse for class-conditional GANs to occur when training data is small (Fig. 2
CONCLUSIONS
In this work, we studied the problem of training class-conditional generative adversarial networks with limited data. Our empirical study demonstrated that class-conditioning can lead the training of GANs to mode-collapse within the investigated setup. To prevent such collapse, we presented a method of injecting the class conditioning by transitioning from unconditional to the conditional case, in an incremental manner. To enable such transition, we have proposed architectural modifica- tions and training objects, which can be easily adapted by any existing GAN model. The proposed method achieves outstanding results compared the state-of-the-a art methods and established base- lines, for the limited data setup of four benchmark datasets. In the future, we will study our method for other types of conditioning (e.g. semanties and images), well as other architectures.

---

Published as a conference paper at ICLR 2022
REFERENCES
Mikolaj Binikowsii, Danica J. Sutherland. Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. European Conference on Computer Vision, 2014.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In IEEE conference computer vision and pattern recognition, pp- 248-255, 2009.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmen- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5356-5364, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. CVPR, 2017.
Minguk Kang and Jaesik Park. ContraGAN: Contrastive Learning for Conditional Image Genera- tion. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems, 2020a.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In Proc. CVPR, 2020b.
Ilya Kavalerov, Wojeiech Czaja, and Rama Chellappa. A study of quality and diversity in K+1 GANs. In Proceedings on "I Can't ' Believe It's Not Better!" at NeurlPS Workshops, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Tech Report, 2009.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. CoRR, 0bbs/19040099) 2019.
Soochan Lee, Junsoo Ha, and Gunhee Kim. Harmonizing maximum likelihood with GANs for multimodal conditional generation. In International Conference on Learning Representations, 2019. URL http:::/
net http///rrrmiidiuxaaaaa.eeeee
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized (gan) training for high-fidelity few-shot image synthesis. In Submitted International Confer- ence on Learning Representations, 2021.
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-iman translation. In IEEE International Conference on Com- puler Vision (ICCV), 2019.
10

---

Published as a conference paper at ICLR 2022
Andreas Lugmayr. Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte. Normalizing flow as a flexible fidelity objective for photo-realistic super-resolution. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 1756-1765, Jan- uary 2022.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv: 1784,,,222200 2014.
arXiv preprint
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In International Conference on Learning Representations, 2018
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze discriminator: A simple baseline for fine-tuning gans. arXiv preprint arXiv:2002. 10964, 2020.
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2750- 2758, 2019.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In International conference on machine learning. pp. 2642-2651, 2017.
Sameera Ramasinghe, Moshiur Farazi, Salman Khan, Nick Barnes, and Stephen Gould. Rethink- ing conditional gan training: An approach using geometrically structured latent manifolds. In NeurlPS, 2021.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Radford Alec, and Chen Xi. Improved techniques for training gans. In Advances in Neural Information Processing Systems, 2016.
Mohamad Shahbazi, Zhiwu Huang. Danda Pani Paudel, Chhatkuli Ajad, and Luc Van Gool. Effi- cient conditional gan transfer with knowledge propagation across classes. IEEE conference on computer vision and pattern recognition, 2021.
Zhangzhang and Song-Chon Cong Zhu. Learning hybrid image templates (hit) by information projec- tion. IEEE Transactions pattern analysis and machine intelligence, 34(7):1354- 1367, 2011.
Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsse -suan Yang, and Weilong Yang. Regularing generative adversarial networks under limited data. In CVPR, 2021.
C. Wah, S. Branson, Welinder, Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. Technical Report ANS-TR-2001-00 California Institute of Technology, 2011.
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bog- dan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision, pp. 218-234, 2018
Yaxing Wang, Abel Gariia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: effective knowledge transfer from gans to target domains with few im- ages. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 9332-9341, 2020.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1316-1324, 2018
Jiahui Yu. Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 2018.
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for limited-data generation. arXiv preprint arXiv:2002./18100 2020a.

---

Published as a conference paper at ICLR 2022
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Advances in Neural Information Processing Systems, volume 33, 2020b.
Zhiming Zhou, Han Cai, Shu Rong, Yuxuan Song, Kan Ren, Weinan Zhang, Jun Wang, and Yong Yu. Activation maximization generative adversarial nets. In International Conference on Learning Representations, 2018.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired maage-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
12

---

In Fig. we provide a detailed visualization of the proposed transition function. As shown, T2 is the starting time and T. the end time of the transition. Note that the end of the training different from the end of the transition. We have added a new term Tm to Fig. 7 to represent the maximum training iterations. First, the model is trained unconditionally from L = until transition to the condition model starts, X going from 0 to linearly. After the end of the transition Published as a conference paper at ICLR 2022
A APPENDIX
In the following, we provide more details, experiments, and visualizations on the discovered obser- vation and the proposed method.
A.1 THE TRANSITION FUNCTION
T.. At Ts the
(Te), the transition function A remains at its maximum value I. In other words, the weight of each loss (of Equation 3 in the paper) does not get adjusted anymore. The end of the transition T. happens at about half the total training time T., in our experiments.
X
d
t- T. T
X
A = 0
(0.
T=
Unconditional Model
Transition
Conditional Model
Figure 5: Visualization of the transition function A, T., T., and Tm denote the start of the transition, the end time of the transition, and the end of the training, respectively.
A.2 MORE DETAILS ON THE IMPLEMENTATION
The proposed transition function makes a transition from to in the specified transition time. However, in the specific case of the AnimalFace dataset, we found that clipping the output of the transition function to the maximum value of 0.2 achieves the best results, which are reported in the main paper.
A.3 EXPERIMENTS ON CIFAR100
In addition to the experiments on the four datasets presented in the paper, we provide the results of training unconditional and conditional StyleGAN2+ADA as well as method, four different subsets of CIFAR100 (Krizhvskyy) (2009) in Table 5 In Fig. 6 as an example, the FID curves of training the three methods subset of CIFAR 100 with 100 classes and 300 images per class are visualized. We observe a similar behavior on CIFAR1O0 where our approach outperforms the conditional baseline, achieving FID and KID better on-par with the unconditional baseline.
13

---

Published as a conference paper at ICLR 2022
Quantitative results for examples of unconditional and conditional training of Style- GAN2+ADA, well as our method, on different subsets of CIFAR100. In the name of the columns, indicates the number classes and shows the number of images per class
Table 5:
Method
C20,
S500
C50, S50
C50,
S300
C100,
S300
FID
KID
FID
KID
FID
KID
FID
KID
UC-StyleGAN+ADA
7
0.0006
20
0.0022
6
0.0008
6
0.0021
C-StyleGAN+ADA
12
0.0033
23
0.0036
9
0.0025
13
0.0053
Ours
7
0.0006
20
0.0014
6
0.0008
6
0.0012
Unconditional
Conditional
Ours
500
100
FID
10
0
5000
120DC
15000
Step
Figure 6: FID curves for training unconditional and conditional StyleGAN2, as well our method, on CIFAR 100 with 100 classes and 300 images per class. The vertical axis is in the log scale.
A.4 CLASS-WISE FID AND KID
Following the common practice (StyleGAN, BigGAN, etc.), we calculate the FID and KID metrics reported our experiments unconditionally, with a class sampling distribution that matches the class distribution of the training dataset. We did not provide the class-wise FID and KID due to the insufficient class-wise sample size. Here in Table we report the blass-wisc metrics for ImageNet Carnivores and Food101, by using all the images of corresponding classes, including the additional images not used for training.
Table 6: The class-wise FID and KID for Imagenet Carnivores and Food101 using the full number of real samples per class in the evaluation.
Loss Formulation
Carnivores
Food101
FID
KID
FID
KID
C-StyleGAN+ADA
139
0.179
100
0.079
C-SYYeGAAN+AAD++PPop
151
0.199
97
0.0815
C-SyyeGAN2+ADAALLccmm
90
0.096
56
0.027
Ours
30
0.011
44
0.019
Note that the values are generally larger for all methods, since a large fraction of FID/KID reference sets were not used for training. However, the relative values still consistent with the metrics ported in the paper. These measures, along with the generated images provided later in the appendix (Figs 92),,, show the class consistency of the proposed method.
14

---

Published as a conference paper at ICLR 2022
A.5 PRECISION AND RECALL
In Table 7 we provide the precision and recall proposed by Kynkäänniemi et al. (019). with the implementation provided by StyleGAN2+ADA. Based the results, unconditional training always yields higher recall, as it can generate between-monl images resulting in bigger diversity. Among the conditional methods, our method yields significantly better recall, while being comparable in terms of precision. Low recall values for the conditional baselines confirm the observed mode collapse. Table 8 we also provide the class-was wiis precision and recall for ImageNet Carnivores and Food101, calculated in the same manner as the class-wise FID and KID in Section A.4
Table 7: The unconditional precision and recall for the compared methods in the paper.
Method
Carnivores Pr
Food101
CUB-200-2011
AnimalFace
R1
Pr
RI
Pr
RI
Pr
RI
UC-StyleGAN2+AAD
0.77
0.300
0.71
0.187
0.70
0.232
0.76
0.430
C-StyleGAN2+ADA
0.80
0,0008
0.74
0.004
0.79
0.002
0.78
0,032
C-StYlcGAAN++ADA+PPPoj
0.81
0.0
0.74
0.002
0.76
0.067
0.83
0.0
C-StyleGAN2++AA+H..ccom
0.81
0.046
0.73
0.004
0.76
0.097
0.83
0.0005
Ours
0.82
0.263
0.82
0.068
0.77
0.229
0.83
0.314
Table 8: The class-wise precision and recall for the compared methods Imagenet Carnivores and Food101 using all of the available real samples per class in the evaluation.
Method
Carnivores
Food101
Pr
RI
Pr
RI
C-StyleGAN2+ADA
0.75
0.0
0.75
0
C-SyleGGAN+AAD++Poojj
0.68
0.0
0.78
0.00005
C-SyleGANN+ADAALLccam
0.67
0.0001
0.70
0.0148
Ours
0.73
0.1205
0.64
0.1049
A.6 MORE ABLATION: STYLEGAN2 WITHOUT ADA
To investigate whether the occurrence of the conditional collapse and efficacy of the proposed method in solving it is related to the adaptive differentiable augmentation (ADA), we perform fur- ther experiments on subset of ImageNet carnivores (50 classes, 500 mages per class), without using ADA in the training. As the FID curves in Fig. and FID and KID scores in Table 9 show, the observed conditioning collapse happens even in the absence ADA. Our method is still able to solve the problem by leveraging the stable behavior of unconditional training
Table 9: The FID and KID scores for training unconditional and conditional StyleGAN2, as well as our method, without using ADA, on ImageNet Carnivores with 50 classes and 500 images per class.
Method
FID
KID
U-StyleGAN2
99
0.0795
C-StyleGAN2
13
0.0067
Ours
8
0.0020
A.7 MORE ABLATION: TRANSITION IN THE LOSS FUNCTION
In Table 10 we provide the ablation over two choices for transition in the proposed loss function:
I. >xa) Teuc + de Tec.
15

---

Published as a conference paper at ICLR 2022
Uncondirional
Conditional
Ours
430
300
FD
200
100
2000
4000
6030
Step
Figure 7: FID curves for training unconditional and conditional StyleGAN2, as well as our method, on ImageNet Carnivores with 50 classes and 500 images per class.
. L + d Le (Proposed in the paper).
The results show better performance for the loss formulation proposed in the paper. Based on the results, the unconditional and conditional training ing. Instead, having the unconditional loss helps the performance. However, as shown in Table of the paper, the two losses seem be conflicting the beginning of the training, resulting in a bad performance in the absence of the transition.
conficting in the later stages of the train-
Table 10: The FID and KID results for two different loss formulations.
Loss Formulation
Carnivores
Food101
CUB-200-2011 KID
AnimalFace FID
FID
KID
FID
KID
FID
KID
L (1 de) Luc d Le
18
0.0047
25
0.0088
25
0.0064
20
0.0035
Line de Lee (Ours)
0.0021
20
0.0045
22
0.0032
16
0.0018
A.8
VISUAL RESULTS
In Fig We provide visual results comparison of our method and baselines. Moreover, more images Randomly generated using method on the four datasets are visualized in Figs. 9 12
16

---

Published as a conference paper at ICLR 2022
UC
Ciiss
BigGAN
Class
Clas
Class
ContraGAN
Class
C-StyleGAN
Class
C-StyleGAN-Prooi
Class
C-tyleGAN+ Lecam
Class
Class Class
Ours
Figure Visual results for the compared methods on four datasets.
17

---

Published as a conference paper at ICLR 2022
Figure 9: Gandomly-generater images using the proposed method trained on ImageNet Carnivores dataset. Each row represents different class. FID score is 14.
18

---

Published as a conference paper at ICLR 2022
wm
Figure 10: Randomly-generated generated images using the proposed method trained on Food101 dataset.
Each row represents a different class. The FID score is 20.
19

---

Published as a conference paper at ICLR 2022
Figure 11: Randomly-generate images using the proposed method trained on CUB-200-2011
dataset. Each row represents a different class. The FID score is 22.
20

---

Published as a conference paper at ICLR 2022
Figure 12: Randomly-geneee generated images using the proposed method trained on the AnimalFace
dataset. Each row represents a different class. The FID score 16.
21


Reviews:
[{"review_id": "kR1MNnL6fQd", "reviewer": "Reviewer_tZA6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The writing of this paper is easy to follow. \n- The idea is novel and intuitive. The authors propose a new training strategy to transfer stylegan2-ada from an unconditional to a conditional approach. And in the results, the best FID was obtained.\n\nWeaknesses & Questions: \n- The reason why this strategy only requires limited data has not been analyzed. Is it the stylegan-ada that reduces the need for data? If so, the \"ada\" strategy is not a contribution of this paper. As the authors claim: \" The proposed method for training cGANs with limited data results not only in stable training…\" . More experiments should be conducted to verify that the proposed method can reduce the need for data. For example, applying the proposed method to stylegan or stylegan2 and training on the same data.\n- Are other architectures supported? The results provided are only implemented on stylegan-ada. However, these statements in Sections 1 and 2 may lead the reader to believe that the proposed approach is a general approach that can be applied to different cGANs architectures. Therefore, in addition to the results of stylegan2-ada, implementations on different architectures should be provided. For example, BigGAN, ContraGAN, etc.\n- More qualitative results are compared. Tables 1 to 4 and Figure 2 provide quantitative results at different settings, however, providing more qualitative results at these settings can help to more visually assess the enhancements from the method.\n\nMinors: \n- Details of L^D_uc, etc. should be specified, and are there additional hyper-parameters for each term?\n- For better illustration, Ts and Te can be added to Figure 1 (first row), although the ranges of Ts and Te can be easily guessed.\n- In Figures 1 and 2, \"our\" outperforms the unconditional version in the FID. Why is the Fid further reduced compared to the unconditional version?\n\n", "summary_of_the_paper": "In this paper, authors proposed a new training strategy which transfer the StyleGAN2-ada to the conditional version gradually by injecting conditional information into the generator and the objective function during the training phase. The proposed method is capable of training on limited data and generating high-quality images. However, the strategy of this method is only applicable to StyleGAN-like architecture and experiments can be improved.", "summary_of_the_review": "This paper provides an intuitive solution to move stylegan2-ada from an unconditional to a conditional approach and achieve the best performance on the FID metric, however more experiments, as listed above, should be given to help a better evaluation of all aspects of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1DRSHG4I942", "reviewer": "Reviewer_juAa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis paper is easy to follow and understand. In my opinion, the authors' observation and proposed training strategy are reasonable. Experiments on four datasets demonstrated the superiority of the proposed strategy. \n\nWeaknesses:\n1) In the paper, the authors claim that \"the class-conditioning training is more robust to data volume reduction\" is an intuitive belief. Based on such intuition, one contribution of this paper is pointing that cGANs are easier to suffer from mode collapse. However, I think this discovery may be widely known by the GAN community. Usually, a stronger condition would lead to worse diversity. For example, conditioning on semantic segmentations would largely decrease the generative diversity.\n\n2) Although the solution/proposed strategy is simple and useful, some principles are not discussed or explored very well. According to experiments, unconditional training seems to correspond to better diversity yet worse quality, conditional training is the opposite. The proposed training seems to find a balance between them. However, is there always the best balance point and how can we arrive there? I believe that function $\\lambda_t$ will largely influence training. For example, if we train cGANs with a large ratio and a relatively long time (e.g., increase $T_e$), will the trained generator collapse again? ", "summary_of_the_paper": "In this paper, the authors work towards training conditional GANs with limited data. Based on the observation that conditional GAN training suffers worse mode collapse than unconditional training, the authors proposed a training strategy that gradually injects conditional information into unconditional training. In other words, a learning curriculum that gradually replaces unconditional GANs training (losses and structure) with conditional ones is proposed.\n\nOverall, the proposed learning curriculum is reasonable and achieved promising performance on data-efficient cGANs training.", "summary_of_the_review": "Overall, the authors work on an interesting topic and present promising results.  In my opinion, the solution seems reasonable, yet there are still many problems that should be further explored. I highly suggest the authors further improve this work. However, I may not champion the current version. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0curQshYD19", "reviewer": "Reviewer_F98y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n[S1] The authors provide interesting observations of cGANs vs unconditional GANs in limited data settings which have not been observed or reported in previous works.\n\n[S2] Comprehensive experiments are conducted to demonstrate the claim that class-conditioning causes mode collapse in limited data settings, whereas unconditional learning leads to satisfactory generative ability.\n\n[S3] The authors propose a simple yet effective method for training cGANs that effectively prevents the observed mode-collapse and results in significant empirical improvements compared to state-of-the-art methods and established baselines.\n\n[S4] The authors also perform a comprehensive ablation analysis over different components of their method.\n\n\nWeaknesses:\n\n[W1] The authors incorporate their method only in StyleGAN2 with ADA. While I understand that it is a recent state-of-the-art method for unconditional and class conditional image generation under a limited-data setup, it would be great if authors can verify the efficacy of their approach when applied to different models like BigGAN, etc.\n\n[W2] The authors do not compare with some of the datasets that a recent method uses - Regularizing Generative Adversarial Networks under Limited Data (Tseng et al. 2021). Tseng et al. also perform experiments with GANs where they whittle down training data and they primarily conduct experiments with CIFAR10 and CIFAR100. The authors do not show their observations and experiments on C10 and C100. It would be nice to have results on these two datasets.\n\n[W3 minor] The technical novelty of the paper is not much. Similar strategies of transitioning from one objective to include another objective can be seen often. For example, training a model using a pre-text task (contrastive learning, solving jigsaw, etc.) and then finetuning for the target task. Authors follow a similar approach where they kind of pretrain using unconditional learning and later include conditional learning. Having said that, the observations made as well as the application of this simple trick to mitigate mode collapse in cGANs are quite novel (empirically).\n", "summary_of_the_paper": "This work studies the problem of training class-conditional GANs in limited data settings. The authors first empirically demonstrate that class-conditioning results in mode collapse in a limited data regime whereas unconditional learning provided satisfactory generative ability. They perform a comprehensive analysis by gradually reducing the size of the training set in two ways, a) reducing the number of classes while keeping a number of 100 training images per class and b) reducing the number of images per class while using 50 classes in all cases. In both cases, the conditional GAN achieves a better FID for larger datasets while performance deteriorates significantly (experiencing mode collapse) when the dataset size is reduced. On the other hand, the unconditional model achieves consistently better FID in limited data settings. Based on this observation, the authors propose a method of injecting the class conditioning by transitioning from unconditional to the conditional case, in an incremental manner. They delineate the proposed architectural changes and training objectives. Authors base their experiments on StyleGAN2 with adaptive data augmentation (ADA) and four datasets. The empirical results suggest significant improvements compared to state-of-the-art methods and established baselines. The authors also perform a comprehensive ablation analysis to understand the contribution of different components.", "summary_of_the_review": "The authors discover an interesting behavior of cGANs in limited data settings and propose a simple yet very effective method to solve the problem. The authors perform a comprehensive analysis of the problem followed by a comprehensive set of experiments and ablation analysis showing the efficacy of their method. Overall, the paper seems to make important observations and empirical contributions.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-mrfY8zYKfK", "reviewer": "Reviewer_iwi9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The observation is interesting. Indeed, the mode collapse problem happens more frequently in conditional GANs (including other conditional GANs such as image-to-image, tex-to-image generation).\n2. The paper is well-written and easy to follow. The high-level idea and the technical details behind are well-explained. The modification that changes the concatenation to addition operation in the generator makes sense and enables the transition idea.\n\nConcerns:\n1. The transition design for the loss is unclear. 1) The goal is to transit the training from unconditional to conditional settings, 2) the \"no transition\" results in Table 2 shows that the combined loss harms the performance. I am wondering if the authors have tried the transition that the loss function in Equation 3 is set to $L = (1 - \\lambda)L_{uc} + \\lambda L_{c}$. In this case, the weight for the unconditional loss is zero after the transition.\n2. How do the authors determine the subset of classes and images to be used for the experiments?\n3. How do the authors compute the FID and KID scores? Are these scores the average of the per-class ones? Per-class FID/KID scores can verify if the proposed method (unconditional training in the beginning) makes the generator to produce images that does not correspond to the input class label.\n4. Since the authors claim to address the mode collapse issue under limited data setting for conditional GANs. The authors can consider to use the precision and recall scores to better measure the diversity of the generated images, especially the recall score.\nKynkäänniemi et al, \"Improved Precision and Recall Metric for Assessing Generative Models,\" NeurIPS 2019.\n5. What is the total training iteration used for all experiments? According to the paper, $T_s$ is set to 2K while $T_e$ is set to 4K. Is there any case that the best model used for computing the score is from the iteration before the transition ends?\n6. Does the proposed harm the performance if $T_e$ is set to be close to the final training iteration? What are the results of setting $T_e$ to 6K or 7K in Table 4?\n\nSuggestions (not related to rating):\n1. The paper is easy to follow for people familiar with the conditional setting in StyleGAN2. However, the conditional setting is not well-introduced in the original paper which may confuses people when they read this paper. The authors can consider to briefly introduce how conditional generation is achieved in StyleGAN2 (both generator and loss sides) before describing the proposed method.\n2. The authors can consider to use \"clean FID\" score due to the robustness.\nParmar et al., \"On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation.\"", "summary_of_the_paper": "This work address the limited training data issue for class-conditional GANs. Inspired by the observation that unconditional GANs produce more diverse results under the same setting, the authors propose a learning algorithm that transits the training of unconditional toward class-conditional settings. The authors report FID and KID scores on four datasets to verify the proposed approach.", "summary_of_the_review": "Overall, this is a good paper that addresses the limited data issue for conditional GANs. However, I have two major concerns: the transition design for the loss, and evaluation metrics. Please see Weakness section in the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines conditional GANs, which are found to lead to model collapse in low data settings. The paper proposes what appears to be a simple but effective method that addresses the issue. Reviewers were generally happy with the experiments and the utility of the observations and analysis. Code for the method was provided during the author response period. Only one reviewer did not vote to accept this paper, but they did acknowledge that the authors had addressed their concerns during the discussions with the authors. All others rated the paper as an accept.
The AC recommends accepting this paper.
Decision: Accept (Poster)