Searching towards Class-Aware Generators for Conditional Generative Adversarial Networks

1 INTRODUCTION . Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) has attracted considerable attention and achieved great success in image generation . Conditional GAN ( cGAN ) ( Mirza & Osindero , 2014 ) is a type of GAN using class information to guide the training of the discriminator and generator so that it usually obtains a better generation effect . Most cGANs incorporate class information into the generator through Conditional Batch Normalization ( CBN ) ( de Vries et al. , 2017 ) , or into the discriminator through projection discriminator ( Miyato & Koyama , 2018 ) , multi-hinge loss ( Kavalerov et al. , 2019 ) , auxiliary loss ( Odena et al. , 2017 ) , etc . In this paper , we investigate the possibility of designing class-aware generators for cGAN ( i.e. , using a distinct generator network architecture for each class ) . To automatically design class-aware generators , we propose a neural architecture search ( NAS ) algorithm on top of reinforcement learning so that the generator architecture of each class is automatically designed . However , as the number of classes increases , there are three main issues we have to consider . First , the search space will grow exponentially as the number of categories grows ( i.e. , combinatorial explosion ) . Second , training the generator separately for each class is prone to insufficient data ( Karras et al. , 2020 ) . Furthermore , searching and re-training each generator one by one may be impractical when the number of generators is large . We propose solutions for these challenges . First , we present a carefully designed search space that is both flexible and safe . We refer to flexibility as the ability to assign a distinct generator architecture to each class , which makes the search space exponentially large while its size is still controllable . To guarantee the safety ( i.e. , enable the limited amount of training data to be shared among a large number of generators ) , we introduce a new operator named Class-Modulated convolution ( CMconv ) . CMconv shares the same set of convolutional weights with a regular convolution but is equipped with a standalone set of weights to modulate the convolutional weights , allowing the training data to be shared among different architectures and thus alleviating the inefficiency on training data . Second , to make the procedure of search and re-training as simple as possible , we develop mixed-architecture optimization , such that the training procedure of multiple class-aware generators is as simple as that of training only one generator . Integrating these modules produces the proposed Multi-Net NAS ( MN-NAS ) . To the best of our knowledge , this is the first method that can produce a number of generator architectures , one for each class , through one search procedure . Figure 1 shows the overall framework of MN-NAS . It applies a Markov decision process equipped with moving average as the top-level logic for sampling and evaluating candidate architectures . After the search procedure , the optimal architecture for each class is determined and they get re-trained and calibrated for better image generation performance . We perform experiments on some popular benchmarks , including the CIFAR10 and CIFAR100 datasets that have different numbers of classes . We achieve FID scores of 5.85 and 12.28 on CIFAR10 and CIFAR100 respectively , which are comparable to state-of-the-art results . In addition to achieving good performance , our method has given us some inspiration . For example , we find the phenomenon that the coordination between the discriminator and generator is very important ( i.e. , to derive distinct class-aware generators , the discriminator must also be class-aware ) . More interestingly , by analyzing the best model found by NAS , we find that the class-modulated convolution is more likely to appear in the early stage ( close to the input noise ) of the generator . We think this phenomenon is related to the semantic hierarchy of GANs ( Bau et al. , 2018 ; Yang et al. , 2020 ) . We apply this finding as an empirical rule to BigGAN ( Brock et al. , 2018 ) , and also observe performance gain . This implies that our algorithm delivers useful and generalized insights to the design of cGAN models . We will release code and pre-trained models to facilitate future research . 2 RELATED WORK . Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) have demonstrated impressive generation capabilities ( Karras et al. , 2017 ; Brock et al. , 2018 ; Karras et al. , 2019a ) . Nevertheless , it has notorious issues like vanishing gradient , training instability , and mode collapse . There are a number of improvements for the original GAN , e.g. , changing the objective function ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Mao et al. , 2016 ; Jolicoeur-Martineau , 2019 ; Qi , 2017 ) , improving network architecture ( Radford et al. , 2015 ; Brock et al. , 2018 ; Karras et al. , 2019a ; Denton et al. , 2015 ; Zhang et al. , 2018 ; Karnewar & Wang , 2019 ) , using multiple generators or discriminators ( Tolstikhin et al. , 2017 ; Hoang et al. , 2018 ; Arora et al. , 2017 ; Durugkar et al. , 2017 ; Ghosh et al. , 2018 ; Nguyen et al. , 2017 ) . Recently , the surge in neural architecture search ( NAS ) has triggered a wave of interest in automatically designing the network architecture of GAN ( Wang & Huan , 2019 ; Gong et al. , 2019 ; Tian et al. , 2020b ; Gao et al. , 2019 ; Tian et al. , 2020a ; Li et al. , 2020 ; Fu et al. , 2020 ; Kobayashi & Nagao , 2020 ) . Conditional GAN ( cGAN ) ( Mirza & Osindero , 2014 ) is another type of GAN that incorporates class information into the original GAN , so that achieving promising results for the class-sensitive image generation task . Most of the early methods just incorporated the class information by concatenation ( Mirza & Osindero , 2014 ; Reed et al. , 2016 ) . AC-GAN ( Odena et al. , 2017 ) incorporated the label information into the objective function of the discriminator by an auxiliary classifier . Miyato & Koyama ( 2018 ) proposed the class-projection ( cproj ) discriminator , which injected class information into the discriminator in a projection-based way . Furthermore , conditional batch normalization ( CBN ) ( de Vries et al. , 2017 ) is a very effective method to modulate convolutional feature maps by conditional information . Subsequently , cproj and CBN are widely used together , forming some powerful cGANs for class image generation ( Zhang et al. , 2018 ; Brock et al. , 2018 ) . 3 OUR APPROACH . We use NAS to design class-aware generators for cGAN . However , implementing this cGAN model is not a trivial task . First , it is not easy to define the search space and the generator of each class may suffer insufficient training data . To tackle these issues , we detail the search space and a weight sharing strategy in Section 3.1 . Second , we must design an efficient search method . In Section 3.2 , we introduce the Multi-Net NAS ( MN-NAS ) and the mixed-architecture optimization , these methods making the procedure of search and re-training of multiple networks simple . 3.1 SEARCH SPACE : SHARING DATA BY CLASS-MODULATED CONVOLUTION . The design of the search space follows the popular cell-based style . The input latent vector is passed through L up-sampling layers , each of which is followed by a cell that does not change the shape ( spatial resolution and channel number ) of the data . Each contains an input node , an output node , and N intermediate nodes and there exists an edge between each pair of nodes , propagating neural responses from the lower-indexed node to the higher-indexed node . Each node summarizes inputs from its precedents , i.e. , xj = ∑ i < j oi , j ( xi ) where oi , j ( · ) is the operator on edge ( i , j ) , chosen from the set of candidate operators , O . To guarantee that the shape of data is unchanged , at the beginning of each cell , the data is pre-processed using a 1 × 1 convolutional layer that shrinks the number of channels by a factor of N . Hence , the output of intermediate nodes , after being concatenated , recover the original data shape . An architecture with tentative operators is shown in Figure 2 . Since the operator used in each edge can be searched , the number of different architectures is |O|L× ( N+1 2 ) . Note that we allow each class to have a distinct architecture , therefore , if there are M classes in the dataset , the total number of possible combinations is |O|L× ( N+1 2 ) ×M . This is quite a large number . Even with the simplest setting used in this paper ( i.e. , |O| = 2 , L = 3 , N = 2 ) , this number is 290 ≈ 1.2× 1027 for a 10-class dataset ( e.g. , CIFAR10 ) or 2900 ≈ 8.5× 10270 for a 100-class dataset ( e.g. , CIFAR100 ) , much larger than some popular cell-based search spaces ( e.g. , the DARTS space ( Liu et al. , 2019 ; Chen et al. , 2019 ; Xu et al. , 2020 ) with 1.1×1018 architectures ) . Class-Modulated Convolution . The first challenge we encounter is that the training data need to be distributed among all classes . That being said , if the architectures of all classes are ‘ truly ’ independent ( e.g. , all network weights are individually optimized ) , the amount of training data for each architecture , compared to the scenario that all classes share the same architecture , is reduced by a factor of M . This can result in severe over-fitting and unsatisfying performance of image generation . To alleviate this risk , we share training data among different architectures by reusing model weights . In practice , most network weights are contributed by the convolutional layers , so we maintain one set of convolutional kernels and use a light-weighted module to introduce class-conditional information . Inspired by CBN ( de Vries et al. , 2017 ) and the ‘ demodulation ’ operation ( Karras et al. , 2019b ) , we propose the Class-Modulated convolution ( CMconv ) operator to incorporate classconditional information . As shown in Figure 3 , a CMconv layer consists of three parts , modulation , demodulation , and convolution . The CMconv shares convolutional weights with the corresponding regular convolution ( Rconv ) . Mathematically , let x denote the input features maps with a class label of y , and ω represent the weights of convolution . The goal of modulation is to introduce a scale factor to each of the input channels , i.e. , ω′ = ω sin where both ω and ω′ are in a shape of cin × cout × U . Here , cin and cout are the number of input and output channels , respectively , and U is the kernel size ( e.g. , 3× 3 ) ; sin is a cin-dimensional vector and ω sin multiplies each set of weights ( cout × U numbers ) by the corresponding entry in sin . We follow the conventional formulation of cGAN to define sin as an affine-transformed class vector , i.e. , sin = Aff ( ey ) , where Aff ( · ) is simply implemented as a trainable fully-connected layer and ey is a learnable embedding vector of the class label , y . The goal of demodulation is to normalize the weights and keep the variance of the input and output features same . We follow Karras et al . ( 2019b ) to use ω′′ = ω′ s−1out where sout is a cout-dimensional vector and s−1out indicates element-wise reciprocal ; sout is similar to the ` 2-norm of ω ′ , computed as sout = √∑ cin , u ( ω′cin , · , u ) 2 + , where is a small constant to avoid numerical instability . In summary , Rconv and CMconv start with the same weight , ω , and CMconv modulates ω into ω′′ . Then , regular convolution is performed , i.e. , conv ( x ; ω ) or conv ( x ; ω′′ ) . Since modulation and demodulation introduce relatively fewer parameters compared to convolution , so using weightsharing Rconv and CMconv operators in each edge is a safe option that enables the limited amount of training data to be shared among a large number of generators .

Summary: This paper proposes an interesting idea that adopts NAS to find a distinct architecture for each class based on cGAN framework. Within the framework, the paper also proposes an operator, Class-Modulated convolution (CMconv), to allow the training data to be shared among different architectures, so as to balance the training data across classes. The proposed method leverages a Markov Decision Process (MDP) in the search algorithm, and learns the sampling policy for NAS. Comprehensive experiments demonstrate the class-aware NAS can outperform class-agnostic NAS.