Searching towards Class-Aware Generators for Conditional Generative Adversarial Networks
Authors: Peng Zhou
Lingxi Xie
XIAOPENG ZHANG
Bingbing Ni
Qi Tian
Venue:  (2021)
Keywords: NAS
cGAN

Abstract: Conditional Generative Adversarial Networks (cGAN) were designed to generate images based on the provided conditions, e.g., class-level distributions. However, existing methods have used the same generating architecture for all classes. This paper presents a novel idea that adopts NAS to find a distinct architecture for each class. The search space contains regular and class-modulated convolutions, where the latter is designed to introduce class-specific information while avoiding the reduction of training data for each class generator. The search algorithm follows a weight-sharing pipeline with mixed-architecture optimization so that the search cost does not grow with the number of classes. To learn the sampling policy, a Markov decision process is embedded into the search algorithm and a moving average is applied for better stability. We evaluate our approach on CIFAR10 and CIFAR100. Besides achieving better image generation quality in terms of FID scores, we discover several insights that are helpful in designing cGAN models.

Under review as a conference paper at ICLR 2021
SEARCHING TOWARDS TLASS-AWARE GENERATORS FOR CONDITIONAL GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Conditional Generative Adversarial Networks (cGAN) designed to generate images based on the provided conditions, e.g. class-level distributions, However, existing methods have used the same generating architecture for all classes. This paper presents a novel idea that adopts NAS to find a distinct architecture for each class. The search space contains regular and Blass-mddulated convolutions, where the latter is designed to introduce class-specific information while avoiding the reduction of training data for each class generator. The search algorithm follows a weight-sharing pipeline with mixed-architecture optimization so that the search cost does not grow with the number of classes. To learn the sampling policy, a Markov decision process is embedded into the search algorithm and a moving average is applied for better stability. We evaluate our approach on CIFARIO and CIFAR100. Besides achieving better image generation quality in terms of FID scores, we discover several insights that are helpful in designing cGAN models.
INTRODUCTION
Generative Adversarial Network (GAN) (Goodfellow al.. 2014) has attracted considerable atten- tion and achieved great success in image generation. Conditional GAN (cGAN) (Mirza & Osin- dero 2014) is a type of GAN using class information guide the training of the discriminator and generator so that it usually obtains a better generation effect. Most cGANs incorporate class infor-
mation into the generator through Conditional Batch Normalization (CBN) (de Vries et al. (017). or into the discriminator through projection discriminator (Miyato & Koyama 2018), multi-hinge loss (Kavalerov al. 2019), auxiliary loss (Odena et al. 2017), etc.
In this paper, we investigate the possibility of designing class-aware generators for cGAN (i.e., using a distinct generator network architecture for each class). To automatically design class-aware gener- ators, we propose neural architecture search (NAS) algorithm top of reinforcement learning so that the generator architecture of each class is automatically designed. However, as the number of classes increases, there are three main issues have to consider. First, the search space will grow exponentially as the number of categories grows (i.e., combinatorial explosion). Second, training the generator separately for each class is prone insufficient data (Karras al.. 2020). Further- more, searching and re-training each generator one by one may be impractical when the number of generators large.
We propose solutions for these challenges. First, we present a carefully designed search space that is both flexible and safe. We refer to flexibility as the ability to assign distinct generator architecture to each class, which makes the search space exponentially large while its size is still controllable. To guarantee the safety (i.e., enable the limited amount of training data to shared among large number of generators), we introduce new operator named Modulatd convolution (CMconv). CMconv shares the same set of convolutional weights with a regular convolution but equipped with standalone set of weights to modulate the convolutional weights, allowing the training data to be shared among different architectures and thus alleviating the inefficiency on training data. Second, to make the procedure of search and re-training simple as possible, we develop sixeed-archtecctue optimization, such that the training procedure of multiple class-aware generators as simple as that of training only generator.
Integrating these modules produces the proposed Multi-Net NAS (MN-NAS). To the best of our knowledge, this is the first method that can produce number of generator architectures, one for

---

Under review as a conference paper at ICLR 2021
Class Closs-Agnostie Agnosttic Generator
Class-Aware Aware Generators
Search Space
Search Space
Controller
Sompling
Sampling
Statistics
Training & Evalustion
Optentatios
Training Evoluation
Gain Intuition
Controller
Class-Aware
& Experience
Updating
Controller
Updating
Single-Net RL-based NAS
Multi-Net RL-based NAS
Figure 1: Illustration of Single-Net RL-based NAS and our Multi-Net RL-based NAS. Class-aware generators allow us to discover statistical laws by analyzing multiple network architectures. How- ever, this cannot be achieved by class-agnostic generator because only one generator is searched (i.e., one sample), so we cannot get more information for architecture design.
each class, through one search procedure. Figure [ shows the overall framework of MN-NAS. applies a Markov decision process equipped with moving average as the top-level logic for sampling and evaluating candidate architectures. After the search procedure, the optimal architecture for each class is determined and they get re-traat trainedn and calibrated for better image generation performance.
We perform experiments on some popular benchmarks, including the CIFARIO and CIFARI00 datasets that have different numbers of classes. We achieve FID) scores of 5.85 and 12.28 on CI- FARIO and CIFAR 100 respectively, which are comparable state-of-the-art results. In addition to achieving good performance, phenomenon that the coordination between the discriminator and generator very important (i.e., to derive distinct class-aware generators, the discriminator must also be class-aware). More inter- estingly, by analyzing the best model found by NAS, we find that the class Mlass-modulated convolution is more likely appear in the early stage (close to the input noise) of the generator. We think this phenomenon is related to the semantic hierarchy of GANs ( (Bau et al. 2018 Yang et al. 2020). We apply this finding an empirical rule to BigGAN (Brock et al. 2018), and also observe performance gain. This implies that our algorithm delivers useful and generalized insights to the design of cGAN models. We will release code and pre-trained models facilitate future research.
method has given some inspiration. For example, we find the
RELATED WORK
Generative Adversarial Network (GAN) (Gooffello et al., 2014) have demonstrated impressive
generation capabilities (Karras et al.] 2017 Brock et al. | 2018 Karras et al., (019a)) Nevertheless,
it has notorious issues like vanishing gradient, training instability. and mode collapse. There are a number of improvements for the original GAN, e.g., changing the objective function (Arjovsky)
eet al., 2017; Gulrajani et al. 2017 Mao al. 2016 Jolicoeur Martineau 2019 Qil 2017). im-
proving network architecture (Radford et al. |015 Brock et al. [0188] Karras et al. 2019a| Denton
al.
2015
discrimi- /Zhan al. 2018 Karnewar & Wang 2019)) using multiple generators
nators
Tolstikhin et al. Hoang et al. |018]] Arora et al. 2017; Durugkar et al., 2017 Ghosh]
al.
2018 Nguyen et 2017). Recently, the surge in neural architecture search (NAS) has trig-
gered
a wave of interest in automatically designing the network architecture of GAN (Wang & Huan
2019
Gong et al. 2019; Tian et al. 2020b Gao al. 2019
Tian al. 2020a
Li et al.
2020
et al.
2020 Kobayashi & Nagao 2020).
Conditional GAN (cGAN) (Mirza & Osindero 2014) another type of GAN that incorporates class information into the original GAN, so that achieving promising results for the class-sensitive image generation task. Most of the early methods just incorporated the class information by concatena- tion (Mirza & Osindero 2014; Reed et al., (016). AC-GAN (Odena et al.) (017)) incorporated the label information into the objective function of the discriminator by an auxiliary classifier. MMyato & Koyamal (2018) proposed the class-proeection (cproj) discriminator, which injected class informa- tion into the discriminator in a projection- based way. Furthermore, conditional batch normalization (CBN) éde Vries et al., 2017) ) is very effective method to modulate convolutional feature maps by conditional information. Subsequently, cproj and CBN are widely used together, forming some powerful eGANs for class image generation (Zhang et al. 2018 Brock et al. 2018).
2

---

Under review as a conference paper at ICLR 2021
4x4xCin
4x4xC
8x8xC
16x16xC
32x32xC
32x32x3
Latent vector
Cell
Cell
Cell 2
Image
FC/Reshape
Conv 1x1
Upeample
Upsample
Upsample
B&B
Cell
C N
C
C
HxWx
HxWx
HxW N
HxWx
HxWx
Conv 1=1
2
B
3
#xWxC
In: HxWxC
Out: HxWxC
Figure 2: A tentative architecture in the search space. N stands for the number of nodes in cell, set to as an example. The operators shown in red arrows can be searched. The data shape is unchanged within each cell, so that the number of cells can be arbitrary. This figure is best viewed in color.
OUR APPROACH
We use NAS to design class-aware generators for cGAN. However, implementing this cGAN model not trivial task. First, it is not easy define the search space and the generator of each class may suffer insufficient training data. To tackle these issues, we detail the search space and a weight sharing strategy in Section 3.] Second, we must design an efficient search method. In Section 3.2 we introduce the Multi-Net NAS (MN-NAS) and the mixed-architecture optimization, these methods making the procedure of search and re-training of multiple networks simple.
3.1 SEARCH SPACE: SHARING DATA CLASS-MODULATEED CONVOLUTION
The design of the search space follows the popular cell-based style. The input latent vector is passed through L up-sampling layers, each of which is followed by a cell that does not change the shape (spatial resolution and channel number) of the data. Each contains an input node, output node, and N intermediate nodes and there exists an edge between each pair of nodes, propagating neural responses from the lower-indexed node to the higher-indexed node. Each node summarizes inputs from its precedents, ie., X, = Eies Oz.j (x;) where o,,j (.) is the operator on edge (i,j), chosen from the set of candidate operators, O. To guarantee that the shape of data is unchanged, at the beginning of each cell. the data is pre-processed using the number of channels by a factor of N. Hence, the output of intermediate nodes, after being concatenated, recover the original data shape. An architecture with tentative operators shown in Figure 2
convolutional layer that shrinks
Since the operator used in each edge can be searched, the number of different architectures is |0|= Lx (M1)). Note that allow each class have distinct architecture, therefore, if there M classes in the dataset, the total number of possible combinations is OO Lx =x This is quite a large number. Even with the simplest setting used in this paper (i.e., | = 2, = 3, this number is 290 1.2 202? for 10-class dataset (e.g., CIFAR10) or 2000 ~ 8.5 10270 for a 100-class dataset (e.g., CIFAR 100), much larger than some popular cell-based search spaces (e.g., the DARTS space (Liu al. 2019 Chen al. 2019 Xu et al. 2020 with 1.1 1028 architectures).
2).
Class-Modulan Convolution. be distributed among all classes. That being said, if the architectures of all classes are 'truly' independent (e.g., all network weights are individually optimized), the amount of training data for each architecture, compared to the scenario that all classes share the same architecture, is reduced by factor of M. This can result in severe over- wet-fiiting and unsatisfying performance of image gen- eration. To alleviate this risk, we share training data among different architectures by reusing model weights. In practice, most network weights are contributed by the convolutional layers, so we main- tain one set of convolutional kernels and use a weighted module to introduce condiitonai information. Inspired by CBN (de Vries et al. 2017) and the 'demodulation' operation (Karras ctt al.) (019b). we propose the Clasl-Moduuated convolution ( (CMcomv) operator to incorporate class- conditional information. As shown in Figure 3 CMcony layer consists of three parts, modulation,
The first challenge we encounter is that the training data need
3

---

Under review as a conference paper at ICLR 2021
Class embedding
op
dust.
d...
op:
Concal
dout
Mod
01 dost
Conv
emod
<0>
operation
Op,
Out
Class Modulated Conv
8 [0]
x
Figure 3: A class-modulated convolution (CMconv), where the class-conditional vector (class embedding) used modulate the convolutional weights (shared with the reg- ular convolution).
Pons-hot
Mixed-A Achileccure Optimization
Figure 4: The mixed-architecture optimization with parallelization mini-batch. X stands for cinsorbrooacassinn multiplication, and A repre- sents sum along the second dimension of the input
demodulation, and convolution. The CMconv shares convolutional weights with the corresponding regular convolution (Rconv).
Mathematically, let x denote the input features maps with class label of y. and w represent the weights of convolution. The goal of modulation to introduce a scale factor to each of the input channels, ie., w' = w ( where both w and w' are in a shape of Gm X Cout U. Here, Gin and Cout are the number of input and output channels, respectively, and U is the kernel size (e.g., x 3); Sin is a Cin -dimensional vector and w O Sin multiplies each set of weights (Cout U numbers) by the corresponding entry in Sin- We follow the conventional formulation of cGAN to define Sin as an affine-trance transforedd class vector, i.e., Sin Aff (c,), where Aff (-) is simply implemented trainable fully-come colle-cconeeted layer and ey is a learnable embedding vector of the class label, y. The goal of demodulation is to normalize the weights and keep the variance of the input and output features same. We follow Karras et al. (201bb) to use w" = w' @ Sot where Sout is Coue-dimensional vector and S.i. indicates element-wise reciprocal; Sout is similar to the fz-norm of w', computed as
Sout V Vem..uu (w) + where is small constant to avoid numerical instability.
In summary, Reonv and CMccnv start with the weight, w, and CMconv modulates into w" Then, regular convolution is performed, iee,, conv (x:w) w) or conv wx) Since modulation and demodulation introduce relatively fewer parameters compared to convolution, using weight- sharing Rcon and CMcony operators in each edge is a safe option that enables the limited amount of training data to be shared among large number of generators.
3.2 SEARCH METHOD:: MULTI-NET NAS
The search process is formulated by one-step Markov Decision Process (MDP). We denote a as the action that samples architectures for all classes. Let (a; E (0, 1) be the sampling policy and the learnable parameters, the performance <(a 0) is measured by:
(a [R (a)).
where R (a) the reward function. Throughout this paper, we use the Inception Score as the re- ward. According to REINFORCE (Williams) 1992), the gradient of J (0) with respect to O can be computed as:
VoJ (0) Ear (a((a) - -) Ve log (N (a; 0))
Σ (R(a))-) Vo lolog (N (aki 0)), VI
1
m
where m is the number of sampled architectures and r a baseline reward, set to be the moving average and used to reduce the variance in the training process. We use gradient ascent to maximize (0). Inspired by Cai et al. (2019); Ying et al. (2019), we design a simple policy. We use 0 to denote [O)-dimensional parameter of class and layer so that the probability of sampling
4

---

Under review as a conference paper at ICLR 2021
each operator, Prob (o)00,)), is determined by the softmax output of 01,s. Hence, given class aware architectures, Aca. the probability that it gets sampled Prob (Aca) т (a; 0)) II., Prob (o)01.)).
3.2.1 OVERALL PIPELINE: SEARCH, RE-TRAINING, AND CALIBRATION
There are two sets of parameters to be learned, ie.. 0 for the sampling policy and w for the super- network. We use the hinge adversarial loss (Lim & Ye 2017)
Ln = gggw) (Eg(its) [max {0, 1- (x, wHll + By(s) (ppp)) [max 1+ ((G w/]].
CG E Be()) [pp()) [D (G (z.y) y) v)]].
where x with class label y is sampled from the real dataset, (-) and (.) denote the discriminator and generator, respectively, and (z) is the standard Gaussian distribution. We use the discriminator in AutoGAN (Gong et al. (019) and add class projection (cproj (Miyato & Koyama 2018)) to it.
We emphasize that using class-aware discriminator is critical to algorithm (please refer to Section 4.2.1).
During the search phase, we adopt the weight-sharing NAS approach (Pham al. 2018) optimize the generator, i.e., the super-network parameterized by w. We perform fair sampling strategy (Chu al., 2019) offer equal opportunity for training each generator architecture. After every Teritic 5 iterations, we update the generator weights; after every polliy = 50 iterations, we update the policy parameters, O. The pseudo code of the optimization process is provided in Appendix A
After the search, we obtain the generator architecture for each class by choosing the operator with the largest score on each edge, i.e., 0t,k = arg max, Prob (o)0).x). Then, we re-irain the gener- ator from scratch following the same procedure. The last step is named calibration, in which we fine-tune the each architecture on the corresponding class for a small number of iterations (thus the overhead is small). As we will show in experiments, the calibration step largely boosts the perfor- mance, because the re-training stage has pursued for the optimality over all classes, which does not necessarily align with the optimality each individual class.
3.2.2 SHARING COMPUTATION BY MIXED-ARCHITECTUSS OPTIMIZATION
We notice a technical issue that greatly downgrades the efficiency of both search and re- re-training. Given mini-batch, B, from the training set, B may contain features from multiple classes, they need to be propagated through different architectures. To avoid heavy computational burder"] we propose mixed-architecture optimization to allow different architectures to be optimized within one borvard-then-backward pass in a batch.
The flowehart of mixed-architecture optimization is shown in Figure Let X denote batch of input features of size b x dmn, where is the size of the batch, B, and din is the feature dimensionality. To improve the efficiency of parallelization, X as whole is propagated through every operator, and each training sample chooses the output corresponding to the selected operator. In practice, this is implemented by concatenating all the outputs of (X), E O, and multiply it by P, b |O] indicator matrix. Each row of which is a one-hot, |O] dimeenssil vector indicating the operator selected by the corresponding sample.
Essentially, siixed-architecture optimization performs redundant computation to achieve more effi- cient parallelization. Fach input feature fed into |O] operators, though only one of the outputs will be used. Nevertheless, b x O does not increase with the number of classes and thus our method generalizes well to complex datasets, e.g., CIFARI00.
4 EXPERIMENTS
Dataset and Evaluation. We use CIFARIO and CIFAR100 (Krizhevsky al. 2009) the testbeds. Both datasets have 50,000 training and 10,000 testing images, uniformly distributed over
If we deal with each class in a batch individually, the corresponding architectures need to be loaded to the GPU one by one and the batch size hecomes small, which results in reduction in computational efficiency. Moreover, inefficiency deteriorates with the number of classes increases.
5

---

Under review as a conference paper at ICLR 2021
Table FID scores of class-agnostic and class-aware GANs on CIFARIO.
Intra FIDs
Method
FID
airp.
auto.
cat
deer
dog
frog
horse
ship
NAS-cGAN
13.62
17:32
7.05
NAS-cGAN (calibrated)
26.83
13.05
25.99
21.24
14.56
23.39
17.32
15.17
14.46
14.99
6.63
NAS-caGAN
29.53
12.32
21.83
21.30
16.11
26.61
16.56
16.79
16.43
16.39
6.83
NAS-caGAN (calibrated)
25.36
11.91
22.66
19.63
13.74
23.29
15.81
15.60
13.82
14.78
5.85
NAS-caGAN-light
12.64
24.96
18.62
7.79
NAS-caGAN-light (calibrated)
23.90
12.69
24.13
23.36
18.39
22.12
16.93
14.71
13.17
15.01
6.31
CM
U1
IAGB
Latent vector
Class-agnostic architecture
NA S-CGAN
6FI0 mFL OFLPPS-266))
U
Upsample
Out
Node
A Airplne PS-2.44))
C1: Conv 1x1
CMC
CMC
C3:
Conv 3x3
Automobile (FLOPS=2.53G)
NA N-CaCaGan CaaccGANN AN-light (PTD =6.31, mF1 mFL.OPS=2.36
CMC3:
Clas-Moduulate Conv 3x3
Tr
Bird (FL.PPS-2236))
Figure 5: Part of the generator architectures found by NAS-caGAN-iigh on the CIFARIO dataset. Compared to the class-agnostic architecture (shown at the top), the class-aware architectures enjoy lower FID score as well cheaper computation. This figure is best viewed in color.
10 or 100 classes. We use the Inception Score (IS) (Salimans al. 2016) and the Fréchet Incep- tion Distance (FID) (Heusel et al. 2017) to measure the performance of GAN on 50K randomly generated images. The FID statistic files pre-caleulated using all training images. We also com- pute the FID score within each class of CIFAR10. Specifically, for each class, we first use 5K real training images pre-calculate the statistic file, and then randomly generate another 5K images for computing the intra FID score. For more experimental details, please refer to Appendix B
4.1 QUANTITATIVE RESULTS
Table summarizes the image generation results on CIFAR10. We use NAS-cGAN to denote the method that each class uses the same searched generator architecture. and MAS-caGAN denote the method with class-aware generators incorporated. The former option is achieved by modified version of the proposed search algorithm that the sampling policy is shared among all classes. It can be seen that NAS-caGAN produces lower FID scores, indicating its better performance compared to NAS-cGAN. After calibration, MAS-caGAN achieves even better results, reporting an FID of 5.85. We also compare the calibrated versions of NAS-cGAN and NAS-caGAN and find that the latter better, indicating that both class-aware architectures and calibration contribute to the generating better images.
We notice that the current space chooses each operator between RConv and CMConv both which parameterized and expensive. To find computationally efficient architectures, we add the op- erator into the search space, and without any further modification, derive a weightee version of class-award generators, denoted NAS-caGAN-light With without calibration, NAS-caGAN- light achieves comparable FID values, while enjoying a reduced average computational overhead (2.34G FLOPs 2. 2.56G FLOPs) of NAS-caGAN. This is another merit of using class-aware

---

Under review as a conference paper at ICLR 2021
CMC3
CMC3
CMC3
89%
RC3
RC3
15%
10%
Layer
Layer
Layer
(a)
(b)
(c)
Figure 6: The proportion of RConv and CMConv in each layer of the searched class-aware archi- tectures, where (a) is obtained using normal discriminator CIFARIO, (b) is obtained using cproj discriminator on CIFARIO, and (c) is obtained using a cproj discriminator on CIFARIO0. See
Section 4.2 for details.
chitectures. Figure shows part of the generator architectures found by NAS-caGAN-ligh. More architecture details are shown in Appendix C Please refer to Appendix C.2 for comparison results with other cGAN models, and to Appendix for the unconditional image generation experiment.
Next, we challenge our method by evaluating on CIFARI00 which has much more classes. Thanks to the proposed mixed-arcee aecee--cchiiecture optimization, we can perform architecture search on CIFAR without additional engineering efforts compared to that on CIFARIO. Differently, we do not perform calibration or evaluate the intra-class FID scores, since there are only 500 training images for each class, which is insufficient to approximate the true image distribution.
Still, we use NAS-cGAN and NAS- caGAN to denote the class-agnostic and class-aware versions of cGAN, respec- tively. Table 2 summarizes the results. Again, by allowing different classes to have individual generator architec- tures, NAS-caGAN achieves better per- formance in terms of both the FID and IS scores. The searched architectures are shown in Appendix D Though we have not achieved the state-of-the-an re- sults, the idea of designing class-aware generators indeed brings benefits. We believe that our findings can be incorporated into other methods (e.g., Multi-hinge (Kavalerov et al. 2019) FQ-GAN (Zhao et al. 2020 for better performance.
Table 2: FID and IS scores CIFAR100 quoted from the paper.
indicates
Method
FID 4
SN-GAN (Miyato et al. 2018)
18.87
8.19
cproj (Miyato & Koyama 2018)
23.20
9.04
Multi-hinge (Kavalerov al 2019
14.62
13.35
FQ-GAN (Zhao et al. 2020)
8.23
10.62
NAS- (ours) NAS-caGAN (ours)
13.94
8.83
12.28
9.71
4.2 DIAGNOSTIC STUDIES
4.2.1 COORDINATION BETWEEN DISCRIMINATOR AND CLASS-AWARD GENERATORS
Based on the NAS-caGAN model, we investigate the difference between using normal discrimina- tor (i.e., no class information) and using a class-project projection (cproj) discriminator (Miyato & Koyama 2018). On CIFAR10, these models report FID scores of 15.90 and 6.83, respectively, i.e., the cproj discriminator is significantly better. We owe the performance gain to the ability that cproj induces more CMCony operators. As shown in Figure 6a the normal discriminator leads to a sparse use of CMConv operators, while, in Figure 6b CMConv occupies almost half of the operators when cproj is used. That is, the aware generators should be searched under the condition that the discriminator is also class-aware This aligns with the results obtained in (Miyato & Koyama 2018), showing that one usually uses the cproj discriminator together with the generator containing condi- tional batch normalization operations.

---

Under review as a conference paper at ICLR 2021
4.2.2 WHERE SHALL WE PLACE CLASS-AWARD OPERATORS?
Last but not least, we study how our algorithm assigns the class-aware operator (i.e., CMCon)) to different positions of the searched generator architectures. Continuing the previous study, we plot the portion of CMConv on CIFAR 100 in Figure 6c] which shows a very similar trend as the experiments on CIFAR1O. In particular, the portion for the first operator (close to the input noise vector) to use CMConv is 100% on CIFARIO 10 and 85% on CIFARI0O, while the portion for the last operator (close to the output generated image) is only 10% and 19% on CIFARIO 10 and CIFAR 100, respectively. This is to suggest that class information seems very important for capturing the distribution of high-level semantics (close to input), while all classes seem to share similar low-level patterns (close output).
To verify that the finding can indeed enhance GAN models, we experiment on BigGAN (Brock) let al., 2018), manually designed GAN model. 'he generator of BigGAN uses three blocks on CIFARTO, each of which contains two class-conditiona, operations (i.e., CBN) by default. We try to put the CBN in different blocks to study the relationship between accuracy and injection position of class information. As shown in Table 3 the model using CBN only in the first block (index O, close to the input noise) works better than those using CBN in other blocks and even the original BigGAN model (using CBN in all blocks). And using CBN only in the last block (close to output) produces inferior performance. Please refer to Appendix FFor for more details.
Next, we study the relationship between accuracy and the po- sition of CMConv by replacing the CBN BigGAN with reg- ular BN, and replacing the regular convolution with CMCom, with other hyperparameters consistent with BigGAN. The re- sults in Table 3 show a similar trend to the experiments of CBN. Besides, the best model using CMCon in the first block is slightly better than the best model of CBN (6.91 vs. 7.11).
Table 3: Evaluating the BigGAN architecture on CIFARIO with CBN (or CMConv) inserted into different blocks. The table is sorted by the 'CMConv' column. Please refer to the main texts for details.
Block index
CBN
CMConv
We explain this phenomenon by the finding (Bau et al. 2018 Yang et al. 2020) that for the generator of GAN, the early and the middle layers determine the spatial layout and the cate- gory attributes, mation, the more benefits; the final layers control the lighting and color scheme of the generated images, which are shared attributes for different classes, so using the regular convolu- tional layer fine. Interestingly, our algorithm offers an alter- native way to reveal this rule which was previously discovered by human experts. This indicates that our method is helpful for understanding and improving existing hand desinne gen- crator network architectures.
0
2
7.11
6.91
the earlier the injection of category infor-
7.39
7.00
v
7.55
7.15
7.28
7.29
8.02
8.06
7.97
8.15
10.60
9.96
CONCLUSIONS
In this paper, we reveal the possibility of designing class-aware generators for conditional GAN models. Though the motivation seems straightforward, non-trivial technical efforts are required to improve the performance as well as efficiency of the algorithm, in particular when the number of classes becomes large. We claim two key contributions. First, the search space containing regular and class midss-moddltted convolutions eases the re- re-training process because all convolutional weights can be trained on the full dataset. Second. we design a miixed-rrchtecture optimization mechanism so that the search and re-training processes can be performed efficiently. In a relatively small search space, the proposed Multi-Net NAS (MN-NAS) algorithm achieves FII) scores of 5.85 and 12.28 on CIFAR10 and CIFAR 100, respectively. Provided more computational resources, our algorithm has the potential of generating images of higher quality.
Our research leaves some open problems to the community. First, we have used the constraint that all operators are either regular or mlass-mddulated convolution. This is to improve the efficiency of utilizing training data, but this also limits the diversity of the searched generator architectures. Second, it is interesting to jointly optimize the architectures of generator and discriminator, since have found strong evidence of their cooperation. We leave these topics to future work.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Net- works. In ICML, 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and Equilibrium in Generative Adversarial Nets (GANs). 1:X3v:770300555 [cs, stat], 2017.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. GAN Dissection: Visualizing and Understanding Generative Adversarial Networks. arXiv:/811.1000 18099 [cs], 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv: 1096, 2018.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct Neural Architecture Search Target Task and Hardware. In ICLR, 2019.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive Differentiable Architecture Search: Bridg- ing the Depth Gap between Search and Evaluation. In ICCV, 2019.
Xin Chen, Lingxi Xie, Jun Wu, Longhui Wei, Yuhui Xu, and Qi Tian. Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks. arXiv:2004.088 08423 [cs, stat], 2020.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search. arXiv: 1907.0185 [cs, stat), 2019.
Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron Courville. Modulating early visual processing by language. In NeurIPS, 2017.
Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. 1xxv:1500.0575 [es]. 2015.
Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative Multi-Adversani Networks. In ICLR, 2017.
Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang- AutoAAN-Distille:: Searching Compress Generative Adversarial Networks. In ICML, 2020.
Chen Gao, Yunpeng Chen, Si Liu, Zhenxiong Tan, and Shuicheng Yan. AdversarialNAS: Adversar- ial Neural Architecture Search for GANs. arXiv:1912.020.0 [cs, eess), 2019.
Arnab Ghosh, Viveka Kulharia, Vinay Namboodiri, Philip H. S. Torr, and Puneet K. Dokania. Multi- Agent Diverse Generative Adversarial Networks. In CVPR, 2018.
Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. AutoGAN: Neural Architecture Search for Generative Adversarial Networks. In ICCV, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In NewrIPS, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In NeurlPS, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to Local Nash Equilibrium. In NewrlPS, 2017.
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. MGAN: Training Generative Adversar- ial Nets with Multiple Generators. In ICLR, 2018
Alexia Jolicoeur-Martinaau. The relativistic discriminator: A key element missing from standard GAN. In ICLR, 2019.
Animesh Karnewar and Oliver Wang. MSG-GAN: Multi-Scale Gradient GAN for Stable Image Synthesis. /xXiv:9903 06048 (cs, stat], 2019.

---

Under review as a conference paper at ICLR 2021
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv:1710.1019 [cs, stat), 2017.
Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. In CVPR, 2019a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and Improving the Image Quality of StyleGAN. arXiv:1912.04958 [cs, eess, stat], 2019b.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training Generative Adversarial Networks with Limited Data. (.XXi::2006.06672 [cs, stat], 2020.
Ilya Kavalerov, Wojciech Czaja, and Rama Chellappa. arXiv:1912.04216 [cs, stat), 2019.
cGANs with Multi-Hinge Loss.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2014.
Masayuki Kobayashi and Tomoharu Nagao. Multi-objective architecture search for generative adversarial networks. In Proceedings of the 2020 Genetic and Evolutionary Computation Con- ference Companion, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun- Yan Zhu, and Song Han. GAN Compression: Efficient Architectures for Interactive Conditional GANs. In CVPR, 2020.
Jae Hyun Lim and Jong Chul Ye. Geometric GAN. arXi::1005.02894 [cond-mat, star], 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. In ICLR, 2019.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. Least Squares Generative Adversarial Networks. 1611:00 [cs], 2016.
Mchdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. arXiv:1411.1784 [cs, stat], 2014.
Takeru Miyato and Masanori Koyama. cGANs with Projection Discriminator. ICLR, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. arXiv:180..05957 [cs, stat], 2018.
Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative adversarial nets. In NeurIPS, 2017.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional Image Synthesis With Aux- iliary Classifier GANs. arX.v:1610.09585 [cs, stat], 2017.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient Neural Architecture Search via Parameter Sharing. In ICML, 2018.
Guo-Jun Qi.
Loss-Sensitive Generative Adversarial Networks on Lipschitz. Densities.
[es], 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, arXiv: 1. 1.06444 [es], 2015.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative Adversarial Text to Image Synthesis. In ICML., 2016.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved Techniques for Training GANs. In NeurlPS, 2016.
10

---

Under review as a conference paper at ICLR 2021
Yuan Tian, Qin Wang, Zhiwu Huang, Wen Li, Dengxin Dai, Minghao Yang, Jun Wang, and Olga Fink. Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search. In ECCV, 2020a.
Yuesong Tian, Li Shen, Li Shen, Guinan Su, Zhifeng Li, and Wei Liu. AlphaGAN: Fully Differ- entiable Architecture Search for Generative Adversarial Networks. arXiv:2006.09134 (cs, eess), 2020b.
Ilya Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabried and Bernhard Schölkopf. AdaGAN: Boosting Generative Models. arXiv: [cs, stat], 2017.
Hanchao Wang and Jun Huan. AGAN: Towards Automated Design of Generative Adversarial Net- works. arXiv: 1080 [cs, stat), 2019.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement
learning, Machine Learning. 8(3), 1992. ISSN 1573-0565, doi: 10. 10.000//FF00992696
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficien architecture search. In ICLR, 2020.
Ceyuan Yang. Yujun Shen, and Bolei Zhou. Semantic Hierarchy Emerges in Deep Generative Rep- resentations for Scene Synthesis. arXiv:1911.09267 [cs], 2020.
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. NAS- Bench-101: 101: Towards Reproducible Neural Architecture Search. arXiv:1902.09635 [cs, stat], 2019.
Han Zhang, lan Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-Attention Generative Adversarial Networks. arXiv:1805.082: 888800088188 [cs, stat], 2018.
Yang Zhao, Chunyuan Li, Ping Yu, Jianfeng Gao, and Changyou Chen. Feature Quantization Im- proves GAN Training. arXiv:2004.02088 [cs, stat], 2020.
11

---

Under review as a conference paper at ICLR 2021
Algorithm Searching with mixed- arcecdetcccttettreee optimization (pseudo-code in PyTorch style).
# Nop
tte number of operators in searched edge
:tee number searched edges in an architecture
# Naritie
the number of times for updating the discriminator per update of the generator :update policy every Npolicy iterations
# Npolicy # No
:tee number of classes
1: for iter, (x. y) in enumerate(data loader):
images æ (b, C. h, w)) class label y: (b,)
# prepare a batch of network architectures
2: fair_arcs
fair_arc_a prr_ssapling() # fair (Nop: L), derived by the fair sampling # ares (b Nop L)
3: ares = fair ceir_arcs.repee repeatt))) 1)
# broadcast inputs to ensure each sample goes through fair arcs
4: a aereeeaat interleave(repeats Nop; dim = 0) # : (b x Nop: G, h, w)
5: y = trepeat_inteellaveerepee
Nop, dim = 0) # (b x Nops)
# update super-network parameters w (we simplify the notation by omitting the noise z)
6: update D(æ, y, arcs)
7: if iter % Neritic == 0:
8:
update Gyy, arcs)
# update policy parameters if iter % poolicy == 0:
9:
# sample generator architectures by the policy for all classes class arcs are are_sammmpppaaaaaa samplingmmnn..... by -polic(() class ares (Ne, L)
11:
reward = eval Inc..iccciionSccocllasssss arcs) update potlteeccccccccccdd
Return: policy T(a;0) 0)
A SEARCH ALGORITHM
Algorithm T shows the pseudo-code of the search procedure. As shown line 2, employ fair sampling strategy (Chu et al. 2019) to offer equal opportunity for training each child model of the super-network. However, unlike FairNAS (Chu al., (209) performing architecture search after the training of the super-network, our method embed the search process (policy learning) into the training loop of the nuper-network (shown on line 12). This is equivalent performing a moving average for the policy parameters over the course of training. Based on NAS-caGAN model, we investigate the difference between these two search strategies. On CIFAR1O, these models report FID scores of 6.83 and 7.61. respectively, i.e., our proposed search strategy is better. We attribute performance gain to the moving average for the policy parameters that could reduce the ranking noise of the weight-sharing NAS (Chen et al. 2020)
Thanks to the proposed nixed-architecture optimization, the search procedure can very simple regardless of the number of classes. As shown on line 8, mixed-architecie optimization allows single forward pass. That to
class image generation with distinct generating architectures say, although each class may have varied generator architecture, these architectures forward in parallel so that the training process is as simple as that the original GAN (Goodfellow et al.] 2014).
B IMPLEMENTATION DETAILS
We use the Adam optimizer Kingma & Ba 2014) with v 0.0001, B1 0. and B2 0.9 for optimizing GAN, and 7 = 0.00035, 31 0.9, and Ba = 0.999 for policy learning. For the discriminator, adopt the same architecture used in AutoGAN (Gong et al.) (2019), but equip it with the cproj (Miyato& & Koyama 2018). The discriminator is updated five times per update the generator. All experiments are performed GeForce GTX-108011 GPU, with the batch size set
12

---

Under review as a conference paper at ICLR 2021
to be 32. We search for 200K iterations and re-train for 500K iterations that are sufficient to achieve stable results.
C SEARCHED GENERATOR ARCHITECTURES ON CIFAR-10
C.I GENERATOR ARCHITECTURES FOR NAS-CGAN AND NAS-CAGAN
We RConv 3 and CMConv 3 x 3 candidate operators. These operators and their corre- sponding index numbers are shown in Table 4 On CIFARIO, there are three cells for generator architecture, each of which contains
nodes, so there are three edges be searched and nine edges in total. The searched generator architectures of IAS-GGAN and NAS- MAS-caGAN are shown in Table 5 It can be seen that the class conditional operator (CMConv) tends to appear the shallow layers (close to the input noise vector), and the class unrelated operator (RConv) prefers to appear at the output layers (close to the generated images). This phenomenon definitely suggestive for future studies on the design of generator architectures.
cell
Table 4: Candidate operators for NAS-cGAN and NAS-caGAN
Index
Operator
RConv x 3 (regular convolution)
1
CMConv 3 x 3 (class-modulated convolution)
Table Searched architectures for NAS-cGAN and NAS-caGAN
Layer
Method
Class
0
2
5
8
NAS-cGAN
All
0
0
1
0
0
1
1
1
0
0
0
0
1
1
1
0
1
1
0
0
4
1
0
1
0
0
1
0
NAS-caGAN
5
1
1
6
1
1
0
1
0
1
0
0
7
1
1
0
8
1
1
0
1
0
0
C.2 COMPARISON WITH STATE-OF-THE-ARE CGANS
Table shows results are not the best, other methods are orthogomal to ours, such as FQ-GAN (Zhao et al.) (2020) and ADA (Karras al. 2020). Combined with other approaches (e.g., FQ-GAN, ADA), our method has the potential to achieve better results.
current state-of-the-art results of cGANs. We emphasize that although
C.3 GENERATOR ARCHITECTURES FOR NAS-CAGAN-LIGHT
We add a non- non-parametee operator, zero, to the search space, and derive a light-weighted version of class-aware generators, named NAS-caGAN-light Table 7 and 8 present the operators with index numbers and searched architectures, respectively. As shown Table g the distribution RCom and CMCony still conforms to the rules mentioned in Sec. C.
aware generator architectures enjoy another merit that different classes could have varied com- putational overhead. Figure shows the computational overhead of each class CIFAR10 10. It can
13

---

Under review as a conference paper at ICLR 2021
Table 6: FID and IS scores on CIFARIO.
indicates quoted from the paper.
Method
FID 4
IS f
SN-GAN (Miyato et al.
2018)
15.73
8.19
eproj (Miyato & Koyama 2018
17.50t
8.62
Multi-hinge (Kavalerov
2019)
6.22
9.55
FQ-GAN (Zhao et al.
2020
6.54
9.18
ADA Karras et al. 2020)
2.67
10.06
NAS-cGAN (ours)
6.63
9.00
MAS-caGAN (ours)
5.85
9.07
be seen that NAS-caGAN is superior to NAS-cGAN in terms of FID score. Both models have almost the same computational overhead for all classes because all the candidate operators are pa- rameterized (RConv and CMConr). After incorporating the non-parameter operator, IAS-caGAN- light achieves comparable FID score with NAS but with less computational overhead. The visualization of the class-aware generators of NAS- IAS-caGAN-light is presented in Figure
Table 7: Candidate operators NAS-caGAN-light
Index
Operator
zero
1
RConv 3 x 3 (regular convolution)
2
CMConv 3 x 3 (class-modulated convolution)
Table 8: Searched architectures for NAS-caGAN-light
Layer
Method
Class
4
8
0
1
2
0
2
2
1
1
2
1
2
2
0
3
2
0
0
1
1
1
1
2
0
NAS-caGAN-light
5
2
0
0
2
2
2
0
0
1
7
0
2
2
8
2
0
2
0
2
2
1
2
14

---

Under review as a conference paper at ICLR 2021
2.6
2.5
2.4
FLOPs (109)
2.3
2.2
2.1
2.0
[FID=6.63] NAS-cGAN
[FID=5.85] NAS-caGAN
1.9
[FID=6.31] NAS-caGAN-light
2
5
6
9
Class
Figure 7: The computational overhead of each class CIFARIO. Under comparable FLOPs, NAS- caGAN is superior to NAS-cGAN in terms FID score. Under comparable FID score, NAS- caGAN-light has fewer FLOPs than NAS-cGAN for each class.
D SEARCHED GENERATOR ARCHITECTURES ON CIFAR-100
We perform experiments CIFARIO0, using the same search space and ryper-parameters in CIFARIO, except increasing the number of classes to 100. That is to say, there will more policy parameters for the search of NAS-caGAN. We emphasize that owing the mixed-architecture optimization, the search and retraining programs can be applied to this setting with more classes without any modification. The candidate operators and searched architectures shown in Table and Table I0 respectively. One can see from Table 10 that the distribution of RConv and CMConv still conforms the rules discussed in Sec.
Table 9: Candidate operators for NAS-cGAN and NAS-caGAN
Index
Operator
0
RConv 3 x 3 (regular convolution)
CMConv33 x 3 (class-modulated convolution)
Table 10: Searched architectures for NAS-cGAN and NAS-caGAN
Layer
Method
Class
0
2
3
4
5
6
7
NAS-cGAN
All
1
0
0
0
NAS-caGAN
0
1
1
0
1
1
0
0
1
0
2
1
1
0
1
0
1
1
0
1
0
1
4
1
1
0
0
5
1
1
0
0
0
0
0
6
1
0
0
0
0
7
1
0
1
1
1
8
0
15

---

Under review as a conference paper at ICLR 2021
10
I
1
1
11
00000000000000
12
1000101
13
14
15
16
17
18
19
20
21
22
23
O
24
25
26
27
10000000
28
29
30
32
33
1
34
35
36
0
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
O
1111111100
0
59
I
111111110
TOOOOOT
60
1
62
O
0
0
63
0
0
0
11111
64
65
0
0
1
1
1001
0
0
66
1
0
16

---

Under review as a conference paper at ICLR 2021
67
1
0000
1
68
0
0
69
1
0
70
1
11
0
71
HODO
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
11111
90
O
92
O
93
94
0
1
0
1
95
0
0
96
0
0
1
0
97
1
1
1
0
0
0
1
98
1
99
0
1
0
O
1
0
0
E UNCONDITIONAL IMAGE GENERATION ON CIFAR 10
We study the effectiveness of the search space and the search algorithm. We de- sign two class-agnostic versions based on the architecture used class-aware setting. The first one is directly set all operators to be RCony, and the sec- ond is search common architecture for all classes, with each the operator on each edge chosen among RConv, skip- connect. and zero. Note that we remove the CMConv operator to guarantee that class information is NOT used at all. The discriminator also class-agnostic with the same architecture used in AutoGAN (without cproj).
Table 11: Comparison between existing automatically
designed GAN methods and our algorithm the setting that all classes share the same generator architecture. No class label is used at all.
Method
Params (M)
FID
IS 1
AGAN (Wang & Huan 2019
20.1
30.50
8:29
0.09
AutoGAN Goong 2019
4.1
12.42
8.55 0.10
AdversarialNAS Gap al. 2019
8.8
10.87
8.74
0.07
baseline (full)
12.26
8.61
0.07
NAS GAN (searched)
5.1
10.80
8.32 0.09
Results are summarized in Table II We deliver two messages. First, our search space that contains 9 selectable operators seems small, but has sufficient ability to find powerful models and compete against recently published methods. Second, the searched NAS-GAN preserves most parameterized operator (RConv), showing that GAN needs sufficient amount parameters achieve good per- formance. This supports the design of our search space, i.e., the competition is held between two parameterized operators. The searched architecture is shown in Table T3
17

---

Under review as a conferrree paper at ICLR 2021
Cal
Coll
00
CMCS
Out
01
C1
Qut
U22
C1
CMC
Out
IRGB
(a) Airplane
Cello
Cell
C:n1 2
00
CMCS
CMCS
Out
U1
CMC3
CMCS
Out
IRGB
(b) Automobile
Catg
0
CANCE Caico
Ouil
In
Qut
U2
5
Out
IRGB
(c) Bird
Caitd di
Coll
Cell 2
10
C1
CMec
Cs
Out
U1
In
Out
U2
In
Our
IRGB
(d) Cat
Call 0
Call
COL 2
U0
C1
CMC
Out
U2
Out
IRGB
(e) Deer
Cell
Gdl
Cell 2
U0
CMLL
0
Out
In
Out
in
CMCS
CMC3 ]
IRGB
(l) Dog
00
CINCS
CNICT
P
11
C1
1all
Out
3
CMCS
Cell
Oul
IRGB
(g) Frog
GdlD
Cell
U0
CMCC
Oui
In
CMC3
Out
UU2
C
In
CMCS
Out
IRGB 1
(h) Horsc
Call
0
C1
CMC
Out
CMC
In
CMC3
CMCS
Out
JRGB
(i) Ship
Cell 0
Call
Cell
01
C
CMCC
C3
Out
UI
C1
In
3
Out
C1
In
Out
IRGB
Fieure 8 Visualization of class Hasss-warre aware generators of (18--aGAA40
Truck
Visualizattoo
T 12: Gandiiatte Operatios for NAS-GAN MAS-GAN
Operratoo
Index
2ero
(iCooma 3 x 3 (regular comovuttion Aagoomnn
2
18

---

Under review as a conference paper at ICLR 2021
30
30
CBN block
CMConv in block
CBN in block
CMConv in black
25
CBN block
25
CMConv in block
block 0,1
CMConv black
CBN block 0,2
20
CMConw in block 0,2
CBN black 1,2
CMConv in black 1.2
0,1.2
CMConv
15
10
10
5
5.
0
0
100
200
300
100
200
300
Epoch
Epoch
(a) CBN
(b) CMCon
Figure 9: Evaluating the effect of the inserted position of CBN (or CMCon)) in BigGAN on CI- FARIO. Inserting CBN (CMConv) in different positions does affect accuracy. This phenomenon is consistent with the rule we summarize through our Multi-Net NAS method.
Table Architectures for baseline and NAS-GAN. The candidate operators and their correspond- ing indices are shown in Table 12
Layer
Method
Class
0
1
2
3
4
5
6
8
baseline (full)
All
2
2
2
2
2
2
2
2
2
NAS-GAN (searched)
All
2
2
2
2
2
F IMPACT OF INJECTION POSITION OF CLASS INFORMATION ON ACCURACY
We use BigGAN, a manually designed GAN model, to study the impact of injection position of class information accuracy. On CIFAR10, the generator of BigGAN has three blocks, each of which contains two class-conditional operations (i.e., CBN). We do controlled experiments by changing the inserted position of CBN (blocks that do hand, we replace the regular convolution in the blocks with CMConv, replace all CBN with regular BN, and study the effect the inserted position of CMCony on the accuracy. As shown in Figure 9 The injection position of class information does have an impact accuracy. This phenomenon has been paid attention by previous work. We discover this phenomenon through our Multi-Net NAS.
CBN use regular BN substitute). On the other
G RESULTS OF CLASS CONDITIONAL IMAGE GENERATION
We show some generated images of NAS-caGAN models trained CIFARIO and CIFAR100, in Figure [00 and Figure [I] respectively. In the figures, each row corresponds to samples of one class. All the samples are randomly sampled rather than cherry-picked.
19

---

Under review as a conference paper at ICLR 2021
Figure 10: Generated images of NAS-caGAN model trained CIFAR10. Each row corresponds to samples of the same class.
20

---

Under review as a conference paper at ICLR 2021
Figure 11: Generated images of NAS-caGAN model trained CIFAR100. Each corresponds to samples of the same class. Due to space limitations, we only show samples of 20 classes without cherry-picking.
21


Reviews:
[{"review_id": "x-pL7mEIRe4", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  \nThis paper proposes to use neural architecture search (NAS) to automatically discover useful conditional generative adversarial network (cGAN) architectures. Specifically, this work aims to find a dedicated architecture for each class.  \n\nStrengths:  \n-Paper is well written.  \n-Demonstrates that optimizing architectures for each class yields some improvement over using a single architecture for all classes.  \n-Architectures learned by the NAS reveal insights about how to use existing building blocks, such as where best to place feature modulation layers in the network.  \n\nWeaknesses:  \n-No random baseline (see [1]).  \n-Given how close NAS-cGAN and NAS-caGAN are in terms of performance, confidence intervals should be reported to confirm that improvement is significant.  \n-Majority of improvement comes from fine-tuning on each class individually. It is unclear how much of this improvement is simply due to additional capacity.  \n-Complexity of the model appears to be disproportionate to the improvement in performance (lots of implementation effort for a somewhat small gain in performance).  \n\nRecommendation and Justification:  \nWhile I think this paper is well written, after reading it I am not convinced of the usefulness of the core idea, which is that generator architectures should be class-aware. It is never explained why it might be desirable for each class to have a distinct generator network, and I cannot think of any reason why this may be the case aside from increased model capacity. For this reason I think that this paper is currently marginally below the acceptance threshold, but look forward to the author's explanation.  \n\nClarifying Questions:  \n-How is calibration performed exactly? Is this procedure the same for NAS-cGAN and NAS-caGAN? Is a separate copy of weights fine-tuned for each class?  If a new copy of weights needs to be created for each class, proper comparison would be to a non-NAS model with up to n_classes times more model capacity than the base model.  \n\n[1] Li, Liam, and Ameet Talwalkar. \"Random search and reproducibility for neural architecture search.\" Uncertainty in Artificial Intelligence. PMLR, 2020.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "me6tDJs9UK8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n \nSummary:\n\nThis paper proposes a framework NAS-caGAN that adopts RL-based NAS to search the optimal class-aware generator architecture by directly optimizing the Inception Score (IS) using the  REINFORCE algorithm, and leverages the mixed-architecture optimization to mitigate the training data sparsity of each category. The authors design a Class-Modulated Convolution to allow for the weight-sharing among different searched architectures. The proposed NAS-caGAN outperforms the model that employs searched class-agnostic architecture on CIFAR 10 and achieves better results compared with cproj (Miyato & Koyama, 2018) on CIFAR 100. \n \n##########################################################################\n \nReasons for score: \n\nOverall, I vote for rejecting, but I am happy to modify the score if the authors could provide further details about my concerns. My major concerns are about the baseline choice, computational costs, and the evidence to support the method’s utility (see cons below).  Hopefully the authors can address my concern in the rebuttal period. \n \n##########################################################################\n \nPros: \n1. The idea of using an RL-based NAS to search for the optimal architecture of different categories is quite interesting and empirically demonstrates its superior effect when compared with the searched identical generator architecture regardless of the category class.\n2. Overall, the paper is well written and technically sound. \n3. Good ablation test that highlights the utility of the class-projection (cproj) discriminator over a standard GAN’s discriminator.\n4. Good illustration of the proportion of different operators (i.e., RConv and CMConv) in each layer of the class-aware architecture.\n \n##########################################################################\n \nCons: \n1. The key concern about the paper is the lack of experiments to validate the utility of the proposed method compared with the previous work. Despite the paper asserting that the usefulness of searching for distinct architecture for different categories, the paper only compares the **searched** class-agnostic model (NAS-cGAN) with the proposed class-aware one (NAS-caGAN), and ignores the comparison with the literature on CIFAR 10, such as (Brock et al., 2018; Kavalerov et al., 2019; Zhao et al., 2020) .\n2.  Another concern is the baseline presented. \n(1) Most recent works such as (Kavalerov et al., 2019; Zhao et al., 2020) selected BigGAN (Brock et al., 2018) as their baseline. Considering the limited results (not achieving the state-of-the-art on neither of the presented metrics), a deeper analysis and performance comparison of the proposed framework would have been helpful to argue for its effectiveness. \n(2) It is interesting to inject CMConv operation at the first layer of BigGAN, which achieves a better result than other operator settings. Why not use BigGAN as the baseline?\n3. Could the authors provide the details about computational and **time** costs of the proposed method? How much time/resources would this method take to search?  \n4. In the Sec.4.1, the paper claims that this method could benefit other works. Considering the lack of convincing experiments to support this argument, it is doubtful about its correctness.\n5. For the “moving average”, it would be better to provide more details about it. The paper claims the usage of it for training stability, but never mentions the details about it, which seems not very clear to me.\n6. In the Sec.2, it would have been nice to supplement the details compared with the recent literature of GANs with NAS.\n \n#########################################################################\n \nMinor comments: \n1. Table 3:  The reported evaluation score is not mentioned, which might be FID.\n2. The “moving average” has not been well addressed in the main paper. \n3. Section 3.1: it would be easier to follow if the authors could paraphrase the last two paragraphs. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VPt_l9eCeut", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\nThis paper designs class-aware generators (increasing flexibility) for cGAN by RL-based neural architecture search (NAS) algorithm. GAN models with better flexibility are promising to yield better performances.\n\nClass-Modulated convolution (CMconv) is designed to increase model flexibility while enabling data sharing among classes, increasing the efficiency of training data.\n\nMixed-architecture optimization is presented to ease the training procedure of multiple class-aware generators.\n\nCons:\n\nThe proposed Class-Modulated Conv also inserts the class embedding to a common convolution, which is similar to a regular BN. The architectures for different categories are still weight-sharing, which is quite a common approach in NAS. Thus, this work may be treated as applying NAS to cGAN.\n\nFor searching network for GANs, the main challenge lies in how to provide stable and efficient supervision as a reward. Note that during training, the generator (G) and discriminator (D) play as rivals. Searching for the architectures of G and choosing IS as a reward only helps the G compete against D. I wonder whether it is the optimal choice, so the authors need to consider the settings in [1], i.e., updating the architectures for both G and D.\n\nMany NAS algorithms for GAN models are relevant to this work, but none of them is evaluated against the proposed method in the experiments. I think the authors should add competing results from AdversarialNAS [1] and AutoGAN. Specifically, the Intra FIDs on CIFAR10, FID, and IS scores on CIFAR100 of the two methods should be reported.\n\nNeither time complexity nor space complexity is reported. Considering that the resolution of outputs is only 32$\\times$32, I wonder whether the proposed method is prohibitively expensive to be applied to practical applications that require higher resolution. Note that the resolution for most generation tasks is at least 256$\\times$256 or 128$\\times$128. I suggest the authors report the training and testing times, as well as the consumed GPU resources.\n\n \n[1] Gao, Chen, Yunpeng Chen, Si Liu, Zhenxiong Tan, and Shuicheng Yan. \"Adversarialnas: Adversarial neural architecture search for gans.\" CVPR, pp. 5680-5689. 2020.\n\n[2] Gong, Xinyu, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. \"Autogan: Neural architecture search for generative adversarial networks.\" ICCV, pp. 3224-3234. 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "L8SETnSWTs3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an interesting idea that adopts NAS to find a distinct architecture for each class based on cGAN framework. Within the framework, the paper also proposes an operator, Class-Modulated convolution (CMconv), to allow the training data to be shared among different architectures, so as to balance the training data across classes. The proposed method leverages a Markov Decision Process (MDP) in the search algorithm, and learns the sampling policy for NAS. Comprehensive experiments demonstrate the class-aware NAS can outperform class-agnostic NAS.\n\n- The paper is well written with sufficient figures and plots. \n- The proposed idea is straightforward and convincing.\n- Rich experiments and analysis are conducted. Implementation details are clearly described in the appendix.\n- I like the figures architecture searched specifically for different classes.\n\nHowever, I still have some concerns:\n\n- From my point of view, the proposed CMconv has exactly the same architecture as the one in Karras et al., instead of using class embedding as input. Please clarify the difference and clearly point out in the paper.\n\n- Quantitative results in Table 2 seem not promising. The proposed method is not compatible with the SOTA. Although the paper claims that the proposed idea can be applied to the existing methods and performs better, there’s no evidence showing that.\n\n- It’d be better to also list infra-FIDs of one existing unconditioned GAN method in the Table 1, for the lower bound / baseline of the experiments.\n\n- For fair comparison, it’s better to have a table with quantitative results of IS/FID on CIFAR-10, listing the existing methods with conditioned/unconditioned, such as AutoGAN, style GAN etc. \n\nOverall, the proposed method is interesting, NAS by MDP with class-aware functionality, which can ideally outperform the class-agnostic based method. The experiments are comprehensive, with strong ablation study and analysis. However, it still requires some clarification and convincing experiments to demonstrate the performance. \n\nI'm willing to rate higher if the concerns are addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6YWwvVDgTj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an interesting method that adopts NAS to search multiple class-aware generator architectures for cGAN instead of class-agnostic type. A search space containing both normal and class-modulated convolutions are introduced to simplify the process of re-training. Besides, this paper design a mixed-architecture optimization to specifically address the computational burden issue under the setting of a multi-net search. The search results also give some insights about constructing cGAN models.\n\n\nStrengths:\n- The perspective of adopting the NAS method to explore the class-aware generator architectures for conditional GANs is relatively novel and interesting, although there are some related works about searching unconditional GANs. \n-The proposed flexibility and safety search space is effective to address the categories grow issue. \n- The developed mixed-architecture optimization is a clever way to improve the efficiency of the search and re-training process.\n- The authors conduct extensive experiments and give some interesting insight/discussion about the results.\n\nWeaknesses:\n- There are no innovative approaches toward neural architecture search are proposed, and this work only focuses on how to bring existing RL-based NAS methods to cGANs while overcoming some issues.\n- The motivation for using distinct architecture to generate images for each class instead of using one architecture is unclear. Most of the existing cGANs in noise-to-image and image-to-image translation settings employ CBN or AdaIN to embed conditional information to a unified generator. \n- The idea is similar to the dynamic routing/inference networks such as Blockdrop [1], and it needs a related discussion about the difference.\n- The experiments are only conducted on low-resolution datasets. In my view, the GANs search algorithm needs to be verified on high-resolution datasets, instead of still continuing to achieve marginal performance improvements on small datasets.\n\nWu, Zuxuan, et al. \"Blockdrop: Dynamic inference paths in residual networks.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the use of class-conditional architectures in GANs. It achieves this by employing neural architecture search (NAS) on top of reinforcement learning. Their main contribution is a “flexible and safe” search space; experiments are carried out on CIFAR-10 and -100. Standard performance results are augmented by diagnostic studies.

This paper received a total of five reviews which, remarkably, yielded the same assessment: the paper had merits but was marginally below the acceptance threshold. In general, the reviewers thought the idea was interesting and straightforward, experiments extensive and the paper was clearly written. The primary concerns brought up were novelty (i.e. just cGAN + NAS?, R1 and R2), minimal performance gain (R4, R5), unclear motivation (R2), lack of comparisons --- including to other NAS for GAN methods --- (R1,R3), limited to low-resolution datasets (R2,R3), no reporting of time or space complexity (R1,R3), unclear where improvement comes from --- no control for capacity --- (R5). 

On the point about comparing to NAS + GAN works, the authors responded, stating that the NAS + GAN methods brought up by the reviewer were unconditional GAN methods and pointed out that they made unconditional GAN comparisons in the Appendix.

The authors also emphasized to multiple reviewers that the point of the paper was not to improve NAS. Interestingly, they also made a comment to R5 that the point of the paper was not to improve performance of cGANs, but to improve understanding of them.	

The reviewers are unanimous in that this paper falls just below the bar for the reasons outlined above. Following the discussion phase, I see no reason to overturn their recommendation. I hope that the authors can use the feedback from these reviews to improve this paper and re-submit it.
Decision: Reject