Collapse by Conditioning: Training Class-conditional GANs with Limited Data

1 INTRODUCTION . Since the introduction of generative adversarial networks ( GANs ) by Goodfellow et al . ( 2014 ) , there has been a substantial progress in realistic image and video generations . The contents of such generations are often controlled by conditioning the process by means of conditional GANs ( cGANS ) ( Mirza & Osindero , 2014 ) . In practice , cGANs are of high interest , as they can generate and control a wide variety of outputs using a single model . Some example applications of cGANs include ; category-conditioned generation ( Brock et al. , 2018 ) , image manipulation ( Yu et al. , 2018 ) , image-to-image translation ( Zhu et al. , 2017 ) , and text-to-image generation ( Xu et al. , 2018 ) . Despite the remarkable success , training cGANs requires large training data , including conditioning labels , for realistic generation and stable training behavior ( Tseng et al. , 2021 ) . Collecting large enough data is challenging in many frequent scenarios , due to the privacy , the quality and the diversity required , among other reasons . This difficulty is often worsened further for datasets for conditional training , where also labels need to be collected . The case of fine-grained conditioning adds an additional challenge for data collection , since the availability of the data samples and their variability are expected to deteriorate with the increasingly fine-grained details ( Gupta et al. , 2019 ) . While training GANs with limited data has recently received some attention ( Karras et al. , 2020a ; Wang et al. , 2018 ; Tseng et al. , 2021 ) , the influence of conditioning in this setting remains unexplored . Compared to the unconditional case , the conditional information provides additional supervision and input to the generator . Intuitively , this additional information can guide the generation process better and ensure the success of cGANs whenever its unconditional counterpart succeeds . In fact , one may even argue that the additional supervision by conditioning can even alleviate the problem of limited data , to an extent . Surprisingly however , we observe an opposite effect ( to our preliminary intuition ) in our experiments for class-conditional GANs . As visualized in Fig . 1 , the conditional GAN trained on limited data suffers from severe mode collapse . Its unconditional counterpart , on the other hand , trained on the same data , is able to generate diverse images of high fidelity with stable training process . To our knowledge , these counter-intuitive observations of cGANs have not been observed or reported in previous works . In this paper , we first study the behaviour of a state-of-the-art class-conditional GAN with varying number of classes and image samples per class , and contrast it to the unconditional case . Our study in the limited data regime reveals that the unconditional GANs compare favorable with conditional ones , in terms of generated image quality . We , however , are interested in the conditional case , so as to be able to control the image generation process using a single generative model . In this work , we therefore set out to mitigate the aforementioned mode collapse problem . Motivated by our experimental observations , we propose a method for training cGANs that leverages the stable training of the unconditional GANs . During the training process , we integrate a gradual transition from unconditional to conditional generative learning . The early stage of the proposed training method favours the unconditional objective for the sake of stability , whereas the later stage favours the conditional objective for the desired control over the output by conditioning . Our transitional training procedure only requires minimal changes in existing state-of-the-art generator and discriminator architectures . We demonstrate the advantage of the proposed method over the existing ones , by evaluating our method on four benchmark datasets under the limited data setups . The major contributions of this study are summarized as follows : • We identify and characterize the problem of conditioning-induced mode collapse when training class-conditional GANs under limited data setups . • We propose a training method for cGANs that exploits the training stability of unconditional GANs to mitigate the observed conditioning collapse . • The effectiveness of the proposed method is demonstrated on four benchmark datasets . The method is shown to significantly outperform the state-of-the art and the compared baselines . 2 CLASS-CONDITIONING MODE COLLAPSE . Learning conditional image generation networks is becoming an increasingly important task . The ability to control the generator is a fundamental feature in many applications . However , even in the context of unconditional GANs , previous studies suggest that class information as extra supervision can be used to improve the generated image quality ( Salimans et al. , 2016 ; Zhou et al. , 2018 ; Kavalerov et al. , 2020 ) . This , in turn , may set an expectation that , the extra supervision by conditioning must not lead to the mode collapse of cGANs in setups where the unconditional GANs succeed . Furthermore , one may also expect to resolve the issue of training cGANs on limited data , to an extent , due to the availability of the additional conditional labels . As the first part of our study , we investigate the effect of conditioning on GANs under the limited data setup . We base our experiments on StyleGAN2 with adaptive data augmentation ( ADA ) , which is a recent state-of-the-art method for unconditional and class conditional image generation under limited-data setup ( Karras et al. , 2020a ) . Both unconditional and conditional versions of StyleGAN2 are trained on four benchmark datasets ( more details in Section 4.1 ) , with the setup of this paper . The selected datasets are somewhat of fine-grained in nature , where the problem of limited data , concerning their availability and labelling difficulty , is often expected to be encountered . In Fig . 1 , we analyze the training of conditional and unconditional GANs by plotting the Fréchet inception distance ( FID ) ( Heusel et al. , 2017 ) during training . The analysis is performed on four different datasets , each containing between 1170 and 2410 images for training . Contrary to our initial expectation , the conditional version obtains consistently worse FID during training , compared to the unconditional training . To analyze its cause , in Fig . 1 we also visualize samples from the best model attained during training in terms of FID . For each dataset , the unconditional model learns to generate diverse and realistic images , while lacking the ability to perform class-conditional generation . On the other hand , the conditional model suffers from severe mode collapse . Specifically , the intra-class variations are very small and mainly limited to color changes , while retaining the same structure and pose . Moreover , the images lack realism and contain pronounced artifacts . Next , we further characterize the mode collapse problem observed in Fig . 1 by analyzing its dependence on the size of the training dataset . To this end , we employ the ImageNet Carnivores dataset ( Liu et al. , 2019 ) , which includes a larger number of classes and images . We perform the analysis by gradually reducing the size of the training set in two ways . In Fig . 2a , we reduce the number of classes while keeping a number of 100 training images per class . In Fig . 2b , we reduce the number of images per class while using a 50 classes in all cases . In both cases , the conditional GAN achieves a better FID for larger datasets , here above 5k images . This observation is in line with previous work ( Salimans et al. , 2016 ; Zhou et al. , 2018 ; Kavalerov et al. , 2020 ) . However , when reducing the dataset size , the order is inverted . Instead , the unconditional model achieves consistently better FID , while the conditional model degrades rapidly . Inspired by these observations , we set out to design a training strategy for conditional GANs that eliminates the mode collapse induced by the conditioning itself . Our proposed approach , presented next , achieves stable training that attains low FID , as visualized in Fig . 1 . The images generated from our model exhibit natural intra-class variations with substantial changes in pose , appearance , and color . Furthermore , our training strategy outperforms both the conditional and unconditional models in terms of FID in a wide range of dataset sizes , as shown in Fig . 2 . 3 METHOD . 3.1 FROM UNCONDITIONAL TO CONDITIONAL GANS . As the analysis in Section 2 reveals , conditional GAN training leads to mode collapse when training data is limited , while the unconditional counterpart achieves good performance for the same number of training samples . This discovery motivates us to design an approach capable of leveraging the advantages of unconditional training in order to prevent mode-collapse in conditional GANs . Our initial inspections of the generated images during conditional training indicated that the mode collapse appears from the very early stages of the training . On the contrary , the corresponding unconditional GAN learns to generate diverse images with gradually improved photo-realism during training . In order to avoid mode-collapse , we aim to exploit unconditional learning in the beginning of the training process . Unconditional training in early stages allows the GAN model to learn the distribution of the real images without the complications caused by conditioning . We then introduce the conditioning in the later stages of the training . This allows the network to adapt to conditional generation in a more stable way , by exploiting the partially learned data manifold . In general , we control the transition between unconditional and conditional generative learning during the training process with a function λt ≥ 0 . The subscript t denotes the iteration number during training . Specifically , a zero value λt = 0 implies a purely unconditional learning , i.e . the conditional information does not affect the generator or discriminator networks . Our goal is to design a training strategy capable of gradually incorporating conditioning during training by increasing the value of λt . While any monotonically increasing function λt may be chosen , we only consider a simple linear transition from 0 to 1 as , λt = min ( max ( t− Ts Te − Ts , 0 ) , 1 ) . ( 1 ) Here , Ts and Te denote the start and end time steps of the transition , respectively . We achieve the desired transition during training by introducing a method for controlling the behavior of the generator and discriminator using the function λt . An overview of our approach , detailed in next sections , is illustrated in Fig . 3 . 3.2 TRANSITION IN THE GENERATOR . The generator G ( z ) of a GAN is trained to map the latent vector z , drawn from a prior distribution p ( z ) , to a generated image Ig = G ( z ) belonging to the distribution of the training data . A conditional generator G ( z , c ) additionally receives the conditioning variable c as input , aiming to learn the image distribution of the training data conditioned on c. The transition from an unconditional G ( z ) to a conditional G ( z , c ) generator seemingly requires a discrete architectural change during the training process . We circumvent this by additionally conditioning our generator on the transition function λt as G ( z , c , λt ) . More specifically , we gradually incorporate conditional information during training by using a generator of the form , G ( z , c , λt ) = G ( S ( z ) + λt · E ( c ) ) . ( 2 ) Here , S and E are neural network modules that transform the latent and conditional vector respectively . In case of λt = 0 , the conditional information is masked out , leading to a purely unconditional generator G ( S ( z ) ) . During the transition step Ts < t < Te , the importance of the conditional information is gradually increased during training . To achieve the generator in equation 2 , we perform a minimal modification to the class-conditional StyleGAN2 ( Karras et al. , 2020b ) baseline . The generator consist of a style-mapping network and an image synthesis network . The style-mapping network maps the input latent vector z and the embedding of the condition c to the intermediate representation w , known as style code . The image synthesis network , then generates images from the style codes . As illustrated in Fig . 3 , we modify the conditioning in the generator by feeding the class embeddings to a fully-connected layer , which is then weighted by λt before adding to the output of the style-mapping network ’ s first layer .

Summary: In this paper, the authors work towards training conditional GANs with limited data. Based on the observation that conditional GAN training suffers worse mode collapse than unconditional training, the authors proposed a training strategy that gradually injects conditional information into unconditional training. In other words, a learning curriculum that gradually replaces unconditional GANs training (losses and structure) with conditional ones is proposed.  Overall, the proposed learning curriculum is reasonable and achieved promising performance on data-efficient cGANs training.