Collapse by Conditioning: Training Class-conditional GANs with Limited Data

1 INTRODUCTION . Since the introduction of generative adversarial networks ( GANs ) by Goodfellow et al . ( 2014 ) , there has been a substantial progress in realistic image and video generations . The contents of such generations are often controlled by conditioning the process by means of conditional GANs ( cGANS ) ( Mirza & Osindero , 2014 ) . In practice , cGANs are of high interest , as they can generate and control a wide variety of outputs using a single model . Some example applications of cGANs include ; category-conditioned generation ( Brock et al. , 2018 ) , image manipulation ( Yu et al. , 2018 ) , image-to-image translation ( Zhu et al. , 2017 ) , and text-to-image generation ( Xu et al. , 2018 ) . Despite the remarkable success , training cGANs requires large training data , including conditioning labels , for realistic generation and stable training behavior ( Tseng et al. , 2021 ) . Collecting large enough data is challenging in many frequent scenarios , due to the privacy , the quality and the diversity required , among other reasons . This difficulty is often worsened further for datasets for conditional training , where also labels need to be collected . The case of fine-grained conditioning adds an additional challenge for data collection , since the availability of the data samples and their variability are expected to deteriorate with the increasingly fine-grained details ( Gupta et al. , 2019 ) . While training GANs with limited data has recently received some attention ( Karras et al. , 2020a ; Wang et al. , 2018 ; Tseng et al. , 2021 ) , the influence of conditioning in this setting remains unexplored . Compared to the unconditional case , the conditional information provides additional supervision and input to the generator . Intuitively , this additional information can guide the generation process better and ensure the success of cGANs whenever its unconditional counterpart succeeds . In fact , one may even argue that the additional supervision by conditioning can even alleviate the problem of limited data , to an extent . Surprisingly however , we observe an opposite effect ( to our preliminary intuition ) in our experiments for class-conditional GANs . As visualized in Fig . 1 , the conditional GAN trained on limited data suffers from severe mode collapse . Its unconditional counterpart , on the other hand , trained on the same data , is able to generate diverse images of high fidelity with stable training process . To our knowledge , these counter-intuitive observations of cGANs have not been observed or reported in previous works . In this paper , we first study the behaviour of a state-of-the-art class-conditional GAN with varying number of classes and image samples per class , and contrast it to the unconditional case . Our study in the limited data regime reveals that the unconditional GANs compare favorable with conditional ones , in terms of generated image quality . We , however , are interested in the conditional case , so as to be able to control the image generation process using a single generative model . In this work , we therefore set out to mitigate the aforementioned mode collapse problem . Motivated by our experimental observations , we propose a method for training cGANs that leverages the stable training of the unconditional GANs . During the training process , we integrate a gradual transition from unconditional to conditional generative learning . The early stage of the proposed training method favours the unconditional objective for the sake of stability , whereas the later stage favours the conditional objective for the desired control over the output by conditioning . Our transitional training procedure only requires minimal changes in existing state-of-the-art generator and discriminator architectures . We demonstrate the advantage of the proposed method over the existing ones , by evaluating our method on four benchmark datasets under the limited data setups . The major contributions of this study are summarized as follows : • We identify and characterize the problem of conditioning-induced mode collapse when training class-conditional GANs under limited data setups . • We propose a training method for cGANs that exploits the training stability of unconditional GANs to mitigate the observed conditioning collapse . • The effectiveness of the proposed method is demonstrated on four benchmark datasets . The method is shown to significantly outperform the state-of-the art and the compared baselines . 2 CLASS-CONDITIONING MODE COLLAPSE . Learning conditional image generation networks is becoming an increasingly important task . The ability to control the generator is a fundamental feature in many applications . However , even in the context of unconditional GANs , previous studies suggest that class information as extra supervision can be used to improve the generated image quality ( Salimans et al. , 2016 ; Zhou et al. , 2018 ; Kavalerov et al. , 2020 ) . This , in turn , may set an expectation that , the extra supervision by conditioning must not lead to the mode collapse of cGANs in setups where the unconditional GANs succeed . Furthermore , one may also expect to resolve the issue of training cGANs on limited data , to an extent , due to the availability of the additional conditional labels . As the first part of our study , we investigate the effect of conditioning on GANs under the limited data setup . We base our experiments on StyleGAN2 with adaptive data augmentation ( ADA ) , which is a recent state-of-the-art method for unconditional and class conditional image generation under limited-data setup ( Karras et al. , 2020a ) . Both unconditional and conditional versions of StyleGAN2 are trained on four benchmark datasets ( more details in Section 4.1 ) , with the setup of this paper . The selected datasets are somewhat of fine-grained in nature , where the problem of limited data , concerning their availability and labelling difficulty , is often expected to be encountered . In Fig . 1 , we analyze the training of conditional and unconditional GANs by plotting the Fréchet inception distance ( FID ) ( Heusel et al. , 2017 ) during training . The analysis is performed on four different datasets , each containing between 1170 and 2410 images for training . Contrary to our initial expectation , the conditional version obtains consistently worse FID during training , compared to the unconditional training . To analyze its cause , in Fig . 1 we also visualize samples from the best model attained during training in terms of FID . For each dataset , the unconditional model learns to generate diverse and realistic images , while lacking the ability to perform class-conditional generation . On the other hand , the conditional model suffers from severe mode collapse . Specifically , the intra-class variations are very small and mainly limited to color changes , while retaining the same structure and pose . Moreover , the images lack realism and contain pronounced artifacts . Next , we further characterize the mode collapse problem observed in Fig . 1 by analyzing its dependence on the size of the training dataset . To this end , we employ the ImageNet Carnivores dataset ( Liu et al. , 2019 ) , which includes a larger number of classes and images . We perform the analysis by gradually reducing the size of the training set in two ways . In Fig . 2a , we reduce the number of classes while keeping a number of 100 training images per class . In Fig . 2b , we reduce the number of images per class while using a 50 classes in all cases . In both cases , the conditional GAN achieves a better FID for larger datasets , here above 5k images . This observation is in line with previous work ( Salimans et al. , 2016 ; Zhou et al. , 2018 ; Kavalerov et al. , 2020 ) . However , when reducing the dataset size , the order is inverted . Instead , the unconditional model achieves consistently better FID , while the conditional model degrades rapidly . Inspired by these observations , we set out to design a training strategy for conditional GANs that eliminates the mode collapse induced by the conditioning itself . Our proposed approach , presented next , achieves stable training that attains low FID , as visualized in Fig . 1 . The images generated from our model exhibit natural intra-class variations with substantial changes in pose , appearance , and color . Furthermore , our training strategy outperforms both the conditional and unconditional models in terms of FID in a wide range of dataset sizes , as shown in Fig . 2 . 3 METHOD . 3.1 FROM UNCONDITIONAL TO CONDITIONAL GANS . As the analysis in Section 2 reveals , conditional GAN training leads to mode collapse when training data is limited , while the unconditional counterpart achieves good performance for the same number of training samples . This discovery motivates us to design an approach capable of leveraging the advantages of unconditional training in order to prevent mode-collapse in conditional GANs . Our initial inspections of the generated images during conditional training indicated that the mode collapse appears from the very early stages of the training . On the contrary , the corresponding unconditional GAN learns to generate diverse images with gradually improved photo-realism during training . In order to avoid mode-collapse , we aim to exploit unconditional learning in the beginning of the training process . Unconditional training in early stages allows the GAN model to learn the distribution of the real images without the complications caused by conditioning . We then introduce the conditioning in the later stages of the training . This allows the network to adapt to conditional generation in a more stable way , by exploiting the partially learned data manifold . In general , we control the transition between unconditional and conditional generative learning during the training process with a function λt ≥ 0 . The subscript t denotes the iteration number during training . Specifically , a zero value λt = 0 implies a purely unconditional learning , i.e . the conditional information does not affect the generator or discriminator networks . Our goal is to design a training strategy capable of gradually incorporating conditioning during training by increasing the value of λt . While any monotonically increasing function λt may be chosen , we only consider a simple linear transition from 0 to 1 as , λt = min ( max ( t− Ts Te − Ts , 0 ) , 1 ) . ( 1 ) Here , Ts and Te denote the start and end time steps of the transition , respectively . We achieve the desired transition during training by introducing a method for controlling the behavior of the generator and discriminator using the function λt . An overview of our approach , detailed in next sections , is illustrated in Fig . 3 . 3.2 TRANSITION IN THE GENERATOR . The generator G ( z ) of a GAN is trained to map the latent vector z , drawn from a prior distribution p ( z ) , to a generated image Ig = G ( z ) belonging to the distribution of the training data . A conditional generator G ( z , c ) additionally receives the conditioning variable c as input , aiming to learn the image distribution of the training data conditioned on c. The transition from an unconditional G ( z ) to a conditional G ( z , c ) generator seemingly requires a discrete architectural change during the training process . We circumvent this by additionally conditioning our generator on the transition function λt as G ( z , c , λt ) . More specifically , we gradually incorporate conditional information during training by using a generator of the form , G ( z , c , λt ) = G ( S ( z ) + λt · E ( c ) ) . ( 2 ) Here , S and E are neural network modules that transform the latent and conditional vector respectively . In case of λt = 0 , the conditional information is masked out , leading to a purely unconditional generator G ( S ( z ) ) . During the transition step Ts < t < Te , the importance of the conditional information is gradually increased during training . To achieve the generator in equation 2 , we perform a minimal modification to the class-conditional StyleGAN2 ( Karras et al. , 2020b ) baseline . The generator consist of a style-mapping network and an image synthesis network . The style-mapping network maps the input latent vector z and the embedding of the condition c to the intermediate representation w , known as style code . The image synthesis network , then generates images from the style codes . As illustrated in Fig . 3 , we modify the conditioning in the generator by feeding the class embeddings to a fully-connected layer , which is then weighted by λt before adding to the output of the style-mapping network ’ s first layer .

Summary: This work studies the problem of training class-conditional GANs in limited data settings. The authors first empirically demonstrate that class-conditioning results in mode collapse in a limited data regime whereas unconditional learning provided satisfactory generative ability. They perform a comprehensive analysis by gradually reducing the size of the training set in two ways, a) reducing the number of classes while keeping a number of 100 training images per class and b) reducing the number of images per class while using 50 classes in all cases. In both cases, the conditional GAN achieves a better FID for larger datasets while performance deteriorates significantly (experiencing mode collapse) when the dataset size is reduced. On the other hand, the unconditional model achieves consistently better FID in limited data settings. Based on this observation, the authors propose a method of injecting the class conditioning by transitioning from unconditional to the conditional case, in an incremental manner. They delineate the proposed architectural changes and training objectives. Authors base their experiments on StyleGAN2 with adaptive data augmentation (ADA) and four datasets. The empirical results suggest significant improvements compared to state-of-the-art methods and established baselines. The authors also perform a comprehensive ablation analysis to understand the contribution of different components.