Year: 2021

Abstract:
Class-conditioning offers a direct means of controlling a Generative Adversarial Network (GAN) based on a discrete input variable. While necessary in many applications, the additional information provided by the class labels could even be expected to benefit the training of the GAN itself. Contrary to this belief, we observe that class-conditioning causes mode collapse in limited data settings, where unconditional learning leads to satisfactory generative ability. Motivated by this observation, we propose a training strategy for conditional GANs (cGANs) that effectively prevents the observed mode-collapse by leveraging unconditional learning. Our training strategy starts with an unconditional GAN and gradually injects conditional information into the generator and the objective function. The proposed method for training cGANs with limited data results not only in stable training but also in generating high-quality images, thanks to the early-stage exploitation of the shared information across classes. We analyze the aforementioned mode collapse problem in comprehensive experiments on four datasets. Our approach demonstrates outstanding results compared with state-of-the-art methods and established baselines.

heading: 1 INTRODUCTION
text: Since the introduction of generative adversarial networks (GANs) by Goodfellow et al. (2014), there has been a substantial progress in realistic image and video generations. The contents of such generations are often controlled by conditioning the process by means of conditional GANs (cGANS) (Mirza & Osindero, 2014). In practice, cGANs are of high interest, as they can generate and control a wide variety of outputs using a single model. Some example applications of cGANs include; category-conditioned generation (Brock et al., 2018), image manipulation (Yu et al., 2018), image-to-image translation (Zhu et al., 2017), and text-to-image generation (Xu et al., 2018).
Despite the remarkable success, training cGANs requires large training data, including conditioning labels, for realistic generation and stable training behavior (Tseng et al., 2021). Collecting large enough data is challenging in many frequent scenarios, due to the privacy, the quality and the diversity required, among other reasons. This difficulty is often worsened further for datasets for conditional training, where also labels need to be collected. The case of fine-grained conditioning adds an additional challenge for data collection, since the availability of the data samples and their variability are expected to deteriorate with the increasingly fine-grained details (Gupta et al., 2019).
While training GANs with limited data has recently received some attention (Karras et al., 2020a; Wang et al., 2018; Tseng et al., 2021), the influence of conditioning in this setting remains unexplored. Compared to the unconditional case, the conditional information provides additional supervision and input to the generator. Intuitively, this additional information can guide the generation process better and ensure the success of cGANs whenever its unconditional counterpart succeeds. In fact, one may even argue that the additional supervision by conditioning can even alleviate the problem of limited data, to an extent. Surprisingly however, we observe an opposite effect (to our preliminary intuition) in our experiments for class-conditional GANs. As visualized in Fig. 1, the conditional GAN trained on limited data suffers from severe mode collapse. Its unconditional counterpart, on the other hand, trained on the same data, is able to generate diverse images of high fidelity with stable training process. To our knowledge, these counter-intuitive observations of cGANs have not been observed or reported in previous works.
In this paper, we first study the behaviour of a state-of-the-art class-conditional GAN with varying number of classes and image samples per class, and contrast it to the unconditional case. Our study in the limited data regime reveals that the unconditional GANs compare favorable with conditional ones, in terms of generated image quality. We, however, are interested in the conditional case, so as to be able to control the image generation process using a single generative model. In this work, we therefore set out to mitigate the aforementioned mode collapse problem.
Motivated by our experimental observations, we propose a method for training cGANs that leverages the stable training of the unconditional GANs. During the training process, we integrate a gradual transition from unconditional to conditional generative learning. The early stage of the proposed training method favours the unconditional objective for the sake of stability, whereas the later stage favours the conditional objective for the desired control over the output by conditioning. Our transitional training procedure only requires minimal changes in existing state-of-the-art generator and discriminator architectures.
We demonstrate the advantage of the proposed method over the existing ones, by evaluating our method on four benchmark datasets under the limited data setups. The major contributions of this study are summarized as follows:
• We identify and characterize the problem of conditioning-induced mode collapse when training class-conditional GANs under limited data setups.
• We propose a training method for cGANs that exploits the training stability of unconditional GANs to mitigate the observed conditioning collapse.
• The effectiveness of the proposed method is demonstrated on four benchmark datasets. The method is shown to significantly outperform the state-of-the art and the compared baselines.

heading: 2 CLASS-CONDITIONING MODE COLLAPSE
text: Learning conditional image generation networks is becoming an increasingly important task. The ability to control the generator is a fundamental feature in many applications. However, even in the context of unconditional GANs, previous studies suggest that class information as extra supervision can be used to improve the generated image quality (Salimans et al., 2016; Zhou et al., 2018; Kavalerov et al., 2020). This, in turn, may set an expectation that, the extra supervision by conditioning must not lead to the mode collapse of cGANs in setups where the unconditional GANs succeed. Furthermore, one may also expect to resolve the issue of training cGANs on limited data, to an extent, due to the availability of the additional conditional labels.
As the first part of our study, we investigate the effect of conditioning on GANs under the limited data setup. We base our experiments on StyleGAN2 with adaptive data augmentation (ADA), which is a recent state-of-the-art method for unconditional and class conditional image generation under limited-data setup (Karras et al., 2020a). Both unconditional and conditional versions of StyleGAN2 are trained on four benchmark datasets (more details in Section 4.1), with the setup of this paper. The selected datasets are somewhat of fine-grained in nature, where the problem of limited data, concerning their availability and labelling difficulty, is often expected to be encountered.
In Fig. 1, we analyze the training of conditional and unconditional GANs by plotting the Fréchet inception distance (FID) (Heusel et al., 2017) during training. The analysis is performed on four different datasets, each containing between 1170 and 2410 images for training. Contrary to our initial expectation, the conditional version obtains consistently worse FID during training, compared to the unconditional training. To analyze its cause, in Fig. 1 we also visualize samples from the best model attained during training in terms of FID. For each dataset, the unconditional model learns to generate diverse and realistic images, while lacking the ability to perform class-conditional generation. On the other hand, the conditional model suffers from severe mode collapse. Specifically, the intra-class variations are very small and mainly limited to color changes, while retaining the same structure and pose. Moreover, the images lack realism and contain pronounced artifacts.
Next, we further characterize the mode collapse problem observed in Fig. 1 by analyzing its dependence on the size of the training dataset. To this end, we employ the ImageNet Carnivores
dataset (Liu et al., 2019), which includes a larger number of classes and images. We perform the analysis by gradually reducing the size of the training set in two ways. In Fig. 2a, we reduce the number of classes while keeping a number of 100 training images per class. In Fig. 2b, we reduce the number of images per class while using a 50 classes in all cases. In both cases, the conditional GAN achieves a better FID for larger datasets, here above 5k images. This observation is in line with previous work (Salimans et al., 2016; Zhou et al., 2018; Kavalerov et al., 2020). However, when reducing the dataset size, the order is inverted. Instead, the unconditional model achieves consistently better FID, while the conditional model degrades rapidly.
Inspired by these observations, we set out to design a training strategy for conditional GANs that eliminates the mode collapse induced by the conditioning itself. Our proposed approach, presented next, achieves stable training that attains low FID, as visualized in Fig. 1. The images generated from our model exhibit natural intra-class variations with substantial changes in pose, appearance, and color. Furthermore, our training strategy outperforms both the conditional and unconditional models in terms of FID in a wide range of dataset sizes, as shown in Fig. 2.

heading: 3 METHOD

heading: 3.1 FROM UNCONDITIONAL TO CONDITIONAL GANS
text: As the analysis in Section 2 reveals, conditional GAN training leads to mode collapse when training data is limited, while the unconditional counterpart achieves good performance for the same number of training samples. This discovery motivates us to design an approach capable of leveraging the advantages of unconditional training in order to prevent mode-collapse in conditional GANs. Our initial inspections of the generated images during conditional training indicated that the mode collapse appears from the very early stages of the training. On the contrary, the corresponding unconditional GAN learns to generate diverse images with gradually improved photo-realism during training. In order to avoid mode-collapse, we aim to exploit unconditional learning in the beginning of the training process. Unconditional training in early stages allows the GAN model to learn the distribution of the real images without the complications caused by conditioning. We then introduce the conditioning in the later stages of the training. This allows the network to adapt to conditional generation in a more stable way, by exploiting the partially learned data manifold.
In general, we control the transition between unconditional and conditional generative learning during the training process with a function λt ≥ 0. The subscript t denotes the iteration number during training. Specifically, a zero value λt = 0 implies a purely unconditional learning, i.e. the conditional information does not affect the generator or discriminator networks. Our goal is to design a training strategy capable of gradually incorporating conditioning during training by increasing the value of λt. While any monotonically increasing function λt may be chosen, we only consider a
simple linear transition from 0 to 1 as,
λt = min
( max ( t− Ts Te − Ts , 0 ) , 1 ) . (1)
Here, Ts and Te denote the start and end time steps of the transition, respectively.
We achieve the desired transition during training by introducing a method for controlling the behavior of the generator and discriminator using the function λt. An overview of our approach, detailed in next sections, is illustrated in Fig. 3.

heading: 3.2 TRANSITION IN THE GENERATOR
text: The generator G(z) of a GAN is trained to map the latent vector z, drawn from a prior distribution p(z), to a generated image Ig = G(z) belonging to the distribution of the training data. A conditional generator G(z, c) additionally receives the conditioning variable c as input, aiming to learn the image distribution of the training data conditioned on c.
The transition from an unconditional G(z) to a conditional G(z, c) generator seemingly requires a discrete architectural change during the training process. We circumvent this by additionally conditioning our generator on the transition function λt as G(z, c, λt). More specifically, we gradually incorporate conditional information during training by using a generator of the form,
G(z, c, λt) = G(S(z) + λt · E(c)) . (2) Here, S and E are neural network modules that transform the latent and conditional vector respectively. In case of λt = 0, the conditional information is masked out, leading to a purely unconditional generator G(S(z)). During the transition step Ts < t < Te, the importance of the conditional information is gradually increased during training.
To achieve the generator in equation 2, we perform a minimal modification to the class-conditional StyleGAN2 (Karras et al., 2020b) baseline. The generator consist of a style-mapping network and an image synthesis network. The style-mapping network maps the input latent vector z and the embedding of the condition c to the intermediate representation w, known as style code. The image synthesis network, then generates images from the style codes. As illustrated in Fig. 3, we modify the conditioning in the generator by feeding the class embeddings to a fully-connected layer, which is then weighted by λt before adding to the output of the style-mapping network’s first layer.

heading: 3.3 TRANSITION IN THE TRAINING OBJECTIVE
text: The unconditional and conditional GAN frameworks also differ in their training objectives. The latter objective assesses the realism of an image based on its conditioning variable. We propose using both unconditional and conditional objectives and follow the same transition as in the generator. Our proposed loss function includes the unconditional objective during the whole training. The conditional term, on the other hand, is weighted by the transition function λt. Our total objective is thus given by,
LD = LDuc + λt · LDc , LG = LGuc + λt · LGc .
(3)
Here, LDuc, L D c , and L D are the unconditional, conditional, and the proposed losses for the discriminator respectively. Similarly, LGuc, L G c , and L
G represent the unconditional, conditional, and the proposed losses for the generator.
As the discriminator is responsible to predict the scores needed for calculating the loss, the proposed training objective requires also modifying the discriminator’s architecture. In this regard, the necessary modification must provide both conditional and unconditional scores. We propose using two prediction branches, separately dedicated to conditional and unconditional cases, in the last layer of the discriminator, as shown in Fig. 3.
The proposed training objective and discriminator for StyleGAN2 are also visualized in Fig. 3. In the discriminator of StyleGAN2, conditioning is performed by matching the features of the input images with the target class embedding. In other words, the conditional discriminator assigns scores
to the input images by calculating the dot-product between the features of the input images and the embedding of the target class c. The proposed discriminator architecture bears some resemblance to the architecture of projection discriminators (Miyato & Koyama (2018)). In contrast to our approach, the projection discriminator aggregates the unconditional and conditional scores inside the discriminator before the loss function. Additionally, the projection discriminator does not perform any transition between the two scores.

heading: 4 EXPERIMENTS
text: In this section, we first provide the details of our experimental setup. Then, we present the quantitative and qualitative results of the proposed method, as well as the comparison with existing methods. Finally, we provide more ablation and analysis of different components of our method.

heading: 4.1 EXPERIMENT SETUP
text: Datasets: We use four datasets to evaluate our method: ImageNet Carnivores (Liu et al., 2019), CUB-200-2011 (Wah et al., 2011), Food101 (Bossard et al., 2014), and AnimalFace (Si & Zhu, 2011). To keep our experiments in the limited-data regime, we decrease the number of classes and images per class in some of these datasets using random sampling.
• ImageNet Carnivores is a subset of the ImageNet dataset (Deng et al., 2009), which contains 149 classes of carnivore animals. The images were processed by the authors to only contain the faces of the animals. We use a subset of the dataset with 20 classes and 100 images per class. • Food101 contains 101 different food categories, having a total amount of 101k images. We use a subset of the dataset with 20 classes and 100 images per class. • CUB-200-2011 contains 200 categories of different bird spices with only around 60 images per class. We use a subset of this dataset with 20 classes with all the images in each class. • AnimalFace, similar to ImageNet Carnivores, is a dataset that contains images of animal faces. However, the animal in AnimalFace are not limited to carnivores. AnimalFace contains 20 classes with 2432 images in total. Since AnimalFace is already a small dataset, we do not further reduce its size.
Implementation details: We base our method on the official PyTorch implementation of StyleGAN2+ADA. The hyper-parameter selection for the base unconditional and conditional StyleGAN2 is performed automatically as provided in the official implementation. Training is done with a batch size of 64 spread over 4 GPUs. For the transition function we use Ts = 2k and Te = 4k.
Evaluation Metrics: We evaluate our method using Fréchet inception distance (FID) as the mostcommonly used metric for measuring the similarity between the distribution of real and generated images. As FID can be biased when real data is small (Karras et al., 2020a), we also include kernel inception distance (KID) (Bińkowski et al., 2018) as a metric which is unbiased by design.

heading: 4.2 RESULTS AND COMPARISONS
text: To assess the efficacy of the proposed method, we provide a quantitative comparison with the wellestablished baselines and existing methods:
• BigGAN+ADA (Brock et al., 2018): Achieving outstanding results on ImageNet, BigGAN has been widely used for class-conditional image generation (more details in Section 5). We use the implementation with ADA provided by (Kang & Park, 2020). • ContraGAN+ADA (Kang & Park, 2020): A class-conditional model based on BigGAN that ouperforms BigGAN in many setups using self-supervision in the discriminator (more details in Section 5). We use the implementation provided by the authors. • U-StyleGAN+ADA: Unconditional training of StyleGAN2+ADA using the original unconditional architecture. • C-StyleGAN+ADA: Conditional training of StyleGAN2+ADA using the original conditional architecture. • C-StyleGAN+ADA+projD: Modified Conditional StyleGAN2+ADA by replacing the discriminator with projection discriminator (Miyato & Koyama, 2018).
• C-StyleGAN+ADA+Lecam: Regularizing C-StyleGAN2+ADA using Lecam regularizer, recently proposed by Tseng et al. (2021) for limited-data setup (more details in Section 5). The authors have suggested a hyper-parameter in the range of [0.1, 0.5]. As we did not observe a noticeable difference between different values in the suggested range, we set the hyper-parameter to 0.3 for all experiments.
Results are reported in Table 1. BigGAN and ContraGAN, even though coupled with ADA, struggle to achieve any good generation quality in our experiments. Conditional StyleGAN2 shows better results on three of the datasets compared to that of the other two conditional competitors. However, the FID and KID scores are still very high. As discussed before, C-StyleGAN is consistently outperformed by its unconditional counterpart. Replacing the StyleGAN2’s discriminator with projection discriminator does not yield a noticeable advantage over the original architecture, as it brings improvements on two of the datasets, but degrades the performance on the other two. Adding Lecam regularization to the C-StyleGAN2 shows promising results, achieving good FID and KID scores on Food101 and AnimalFace. However, it still fails to achieve as good generation quality as the unconditional StyleGAN2. The FID and KID scores for the proposed method indicate significant and consistent advantage over all the compared methods in all four datasets. Our method is able to maintain a stable training and achieve better generation quality than both unconditional and conditional StyleGAN2. The FID curves during training along with example generated images using the
proposed method are visualized in Fig. 1. FID curves indicate training dynamics as stable as the unconditional training, while achieving better FID. In addition, the generated images of our method are clearly of more diversity and quality compared to those of the standard conditional model, showing the advantage of the proposed method. The results in Fig. 2 further demonstrate a clear advantage of our method over a wide range of dataset sizes. Our approach maintains the performance of cGANs for larger datasets, while significantly outperforming the unconditional counterpart when data is more scarce. This shows that our method enables cGANs to use the additional information provided by the labels to achieve better generation quality without falling into any mode collapse induced by the conditioning.

heading: 4.3 ABLATION AND ANALYSIS
text: In this section, we provide further ablation and analysis over different components of our method. First, we provide an ablation study containing four different variants:
• No transition: Training the modified architecture with the new objective, without any transition in the objective or the generator (equivalent to using an auxiliary unconditional loss term to train a conditional model).
• Transition only in G: Performing the transition only in the generator (Sec. 3.2), while the training objective is the summation of the unconditional and conditional term.
• Transition only in Loss: Performing the transition only in the training objective (Sec. 3.3), while the generator is fully conditional from the beginning.
• Final method: The final method with all the proposed components.
Table 2 presents the results of the ablation study on Food101 and ImageNet Carnivores. The No transition version yields poor results, showing that the mode collapse is not alleviated by only adding an auxiliary unconditional training objective. Adding the transition to the generator already brings significant improvement to the model. Having the transition only in the objective, on the other, does not lead to good results. To our initial surprise, this reveals that transitioning in the generator is a crucial part of the
method. However, also having the transition in the objective in addition to that in the generator, corresponding to our final method, achieves the best results.
Next, we analyze the impact of when the transition between unconditional and conditional learning is applied. The total transition time is fixed to Te − Ts = 2k time steps. We then report the results on the Food101 and ImageNet Carnivores datasets for different starting times Ts in Table 3. Importantly, we notice a significantly worse results if the transition is started in the beginning of the training Ts = 0. This further supports the hypothesis that conditioning leads to mode collapse in the early stages of the training. By introducing conditional in-
formation in a later stage, good FID and KID numbers are obtained without being sensitive to the specific choice of Ts. In Table 4, we further independently analyze the transition end time Te, while keeping Ts = 2k fixed. Again, our approach is not sensitive to its value. Our method does thus not require extensive hyper-parameter tuning.
Lastly, we visualize on evolution of the generated images and formation of classes during our training process. Fig 4 shows how different images generated from the unconditional phase of training on AnimalFace all start to evolve into different images of the class Panda. In addition to the formation of the classes, Fig. 4 also shows how the image quality continues to improve during and after the transition. In the Appendix, randomly generated images from all
classes of all datasets are provided for more detailed assessment of the ability of the proposed method in conditional high-quality image generation.

heading: 5 RELATED WORK
text: Class-conditional GANs: The first conditional GAN architecture, introduced by Mirza & Osindero (2014), incorporated conditioning by concatenating the condition variable to the input of the generator and the discriminator. AC-GAN (Odena et al., 2017) equipped the discriminator with an auxiliary classification task to ensure the conditional generation. cGAN with projection discriminator (Miyato & Koyama, 2018) replaced the two-headed discriminator of AC-GAN with a single-output discriminator that outputs the sum of the realism score and class score. The class score is calculated using the dot-product of the discriminator features and the target class embedding. The conditioning in the generator, on the other hand, is performed through the conditional batch normalization layers. Improving over cGAN with projection discriminator, BigGAN (Brock et al., 2018) was able to become the state-of-the-art cGAN on ImageNet dataset (Deng et al., 2009). Inspired by the recent progress in self-supervised learning, ContraGAN (Kang & Park, 2020) improved over BigGAN by exploiting an auxiliary self-supervised task in the discriminator for better image representation learning. StyleGAN (Karras et al., 2019), mainly known for unconditional image generation, was extended to
class conditioning in the improved version, and coupled with adaptive differentiable augmentation, outperformed the state-of-the-art cGANs in small data setup (Karras et al., 2020a).
cGANs in small data regime: There are several directions for addressing the problem of training GANs on small datasets. Transfer learning is one of these directions that exploits a large pretraining dataset to provide a better initialization for learning the target data. In recent years several works have studied the best practices for performing transfer learning in GANs (Wang et al., 2018; 2020; Mo et al., 2020; Zhao et al., 2020a; Noguchi & Harada, 2019)). As an example of transfer learning in the context of conditional GANs, cGANTransfer (Shahbazi et al., 2021) proposed class-specific knowledge transfer from source dataset to the target by learning to combine the source class representations, which embedded as the parameters of the batch normalization layers. Although transfer learning is an effective technique for training GANs on small datasets, it usually requires access to a large pretraining dataset with sufficient domain relevance to the target data. Data augmentation is another technique for tackling the problem of small data. In order to avoid augmentations from leaking to the generated images, recent concurrent works proposed differentiable data augmentation Zhao et al. (2020b); Karras et al. (2020a). Including adaptive differentiable augmentation (ADA) as a part of StyleGAN2 resulted in a significant improvement in conditional image generation quality and training stability on small datasets, outperforming previous conditional GANs on cifar10 dataset (Karras et al., 2020a). In addition to the aforementioned techniques, there are other studies on GANs in small data regime focusing on better architecture or loss function design. Liu et al. (2021) proposed a lighter unconditional architecture for StyleGAN for faster training, as well as a self-supervised discriminator to increase the stability on fewer samples. As another approach, Tseng et al. (2021) proposed a regularization to prevent the discriminator from over-fitting in smalldata setup by penalizing the difference between the predictions of the discriminator for current real images and those of previous generated images, as well as the difference between the predictions for current generated images and those of previous real images.

heading: 6 CONCLUSIONS
text: In this work, we studied the problem of training class-conditional Generative Adversarial Networks (cGANs) with limited data. Our empirical study demonstrated that class-conditioning can lead the training of GANs to mode-collapse within the investigated setup. To prevent from such collapse, we presented a method of injecting the class conditioning by transitioning from unconditional to conditional case, in an incremental manner. To enable such transition, we have proposed architectural modifications and training objects, which can be easily adapted by any existing GAN models. The proposed method achieves outstanding results when compared with the state-of-the-art methods and established baselines, for the limited data setup of four benchmark datasets. In future, we will test and study our method for the cases of conditioning beyond object classes (e.g. semantics).

heading: A APPENDIX
text: In the following, we provide more experiments and visualizations on the discovered observation and the proposed method.

heading: A.1 STYLEGAN2 WITHOUT ADA
text: To investigate whether the appearance of the conditional collapse and efficacy of the proposed method in solving it is related to the adaptive differentiable augmentation (ADA), we perform an experiment on a subset of ImageNet carnivores (50 classes, 500 mages per class), without using ADA in the training. As the FID and KID scores show, the observed conditioning collapse happens even in the absence of ADA. Our method is still able to solve the problem by leveraging the stable behavior of unconditional training

heading: A.2 EXPERIMENTS ON CIFAR100
text: In addition to the experiments on the four datasets presented in the paper, we provide the results of training unconditional and conditional StyleGAN2+ADA, as well as our method, on four different subsets of CIFAR100 (Krizhevsky (2009)) in Table 6. In Fig. 6, as an example, the FID curves of training the three methods on a subset of CIFAR100 with 100 classes and 300 images per class are visualized. We observe a similar behavior on CIFAR100, where our approach outperforms the conditional baseline, achieving FID and KID better or on-par with the unconditional baseline. .

heading: A.3 CLASS-WISE FID AND KID
text: Following the common practice (StyleGAN, BigGAN, etc.), we calculate the FID and KID metrics reported in our experiments unconditionally, with a class sampling distribution that matches the class distribution of the training dataset. We did not provide the class-wise FID and KID due the insufficient class-wise sample size. Here in Table 7, we report the class-wise metrics for ImageNet Carnivores and Food101, by using the full number of images for corresponding classes, including the additional images not used for training.
Note that the values are generally larger for all methods, since a large fraction of FID/KID reference set were not used for training. However, the relative values are still consistent with the metrics reported in the paper. These measures, along with the generated images provided later in the appendix (Figs 9-12), show the class consistency of the proposed method.

heading: A.4 PRECISION AND RECALL
text: In Table 8, we provide the precision and recall proposed by Kynkäänniemi et al. (2019), with the implementation provided by StyleGAN2+ADA. Based on the results, unconditional training always yields a higher recall, as it can generate between-mode images resulting in bigger diversity. Among the conditional methods, our method yields significantly better recall, and comparable in terms of precision. Low recall values for the conditional baselines confirm the observed mode collapse. We also provide the class-wise precision and recall for ImageNet Carnivores and Food101, calculated in the same manner as the class-wise FID and KID in Section A.3.

heading: A.5 THE TRANSITION FUNCTION
text: In Fig. 7, we provide a detailed visualization of the proposed transition function. As shown, Ts is the starting time and Te is the end time of the transition. Note that the end of the training is different from the end of the transition. We have added a new term Tm to Fig. 7 to represent the maximum training iterations. First, the model is trained unconditionally from t = 0 until t = Ts. At Ts, the transition to the condition model starts, λt going from 0 to 1 linearly. After the end of transition (Te), the transition function λt remains at its maximum value of 1. In other words, the weight of each loss (of Equation 3 in the paper) does not get adjusted anymore. The end of transition Te happens at about half the total training time Tm in our experiments.

heading: A.6 MORE ABLATION: TRANSITION IN THE LOSS FUNCTION
text: In table 10, we provide the ablation over two choices for transition in the proposed loss function:
• L = (1− λt) · Luc + λt · Lc.
• L = Luc + λt · Lc (Proposed in the paper).
The results show better performance for the loss formulation proposed in the paper. Based on the results, the unconditional and conditional training are not conflicting in the later stages of the training. Instead, having the unconditional loss actually helps the performance. However, as shown in Table 2 of the paper, the two losses seem to be conflicting in the beginning of the training, resulting in a bad performance in the absence of the transition.

heading: A.7 REDUCED DIVERSITY BY OTHER FORMS OF CONDITIONING
text: Previous studies have reported a decrease in diversity in tasks with strong pixel-level conditioning, such as conditioning on semantic segmentation or on images (e.g. super-resolution) Ramasinghe et al. (2021); Lugmayr et al. (2020). However, these observations does not directly translate to the class-conditional setting. On the contrary, the analysis performed in Fig. 2 shows that the classconditional model achieves consistently better FIDs than the unconditional one if enough data is available (¿5k images). This behavior could therefore be expected to continue when gradually reducing the available training data. In fact, as mentioned in Section 2, some existing works investigate exploiting label supervision to improve unconditional GANs ( (Salimans et al., 2016; Zhou et al., 2018; Kavalerov et al., 2020)). Furthermore, the results obtained by our method in Fig. 2 show that, with the proper way of conditioning, the aforementioned can even be prevented in the limited-data setups, resulting in better FID scores compared to unconditional training.

heading: A.8 OTHER ARCHITECTURES:
text: Initially, we considered both StyleGAN2 and BigGAN as the baselines for our observations and the proposed method. However, the observation and the proposed method are based on setups where unconditional training is stable. BigGAN and BigGAN-based models like ContraGAN are designed and optimized (architecture, hyper-parameters, etc.) only for conditional training. How to transform BigGAN into a capable unconditional architecture has not been explored. We tried to follow the strategy described in Karras et al. (2020a) to design an unconditional BigGAN baseline. However, similar to what Karras et al. (2020a) have reported (Fig 19 in their paper), we were not able to find a stable configuration of unconditional BigGAN in the low-data regime. We therefore also conduct our analysis on the StyleGAN2 architecture. Note that StyleGAN2+ADA represents the state-ofthe-art for both conditional and unconditional generation in the low-data regime. It is therefore the strongest and most relevant baseline for our study. That being said, exploring the observation
and the proposed method in other GAN architectures and other types of conditioning would be an interesting future direction.
A.9 VISUAL RESULTS
We provide more visual results for comparison of our method and the baselines in Fig. 8. Moreover, more generated images using our method on the four datasets are visualized in Figs. 9-12.
Under review as a conference paper at ICLR 2022
C la ss 1 C la ss 2 C la ss 3 C la ss 4 U nc on di tio na l

Review summaries:
In this paper, the authors work towards training conditional GANs with limited data. Based on the observation that conditional GAN training suffers worse mode collapse than unconditional training, the authors proposed a training strategy that gradually injects conditional information into unconditional training. In other words, a learning curriculum that gradually replaces unconditional GANs training (losses and structure) with conditional ones is proposed.  Overall, the proposed learning curriculum is reasonable and achieved promising performance on data-efficient cGANs training.

In this paper, authors proposed a new training strategy which transfer the StyleGAN2-ada to the conditional version gradually by injecting conditional information into the generator and the objective function during the training phase. The proposed method is capable of training on limited data and generating high-quality images. However, the strategy of this method is only applicable to StyleGAN-like architecture and experiments can be improved.

This work studies the problem of training class-conditional GANs in limited data settings. The authors first empirically demonstrate that class-conditioning results in mode collapse in a limited data regime whereas unconditional learning provided satisfactory generative ability. They perform a comprehensive analysis by gradually reducing the size of the training set in two ways, a) reducing the number of classes while keeping a number of 100 training images per class and b) reducing the number of images per class while using 50 classes in all cases. In both cases, the conditional GAN achieves a better FID for larger datasets while performance deteriorates significantly (experiencing mode collapse) when the dataset size is reduced. On the other hand, the unconditional model achieves consistently better FID in limited data settings. Based on this observation, the authors propose a method of injecting the class conditioning by transitioning from unconditional to the conditional case, in an incremental manner. They delineate the proposed architectural changes and training objectives. Authors base their experiments on StyleGAN2 with adaptive data augmentation (ADA) and four datasets. The empirical results suggest significant improvements compared to state-of-the-art methods and established baselines. The authors also perform a comprehensive ablation analysis to understand the contribution of different components.

This work address the limited training data issue for class-conditional GANs. Inspired by the observation that unconditional GANs produce more diverse results under the same setting, the authors propose a learning algorithm that transits the training of unconditional toward class-conditional settings. The authors report FID and KID scores on four datasets to verify the proposed approach.