Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an interesting idea that adopts NAS to find a distinct architecture for each class based on cGAN framework. Within the framework, the paper also proposes an operator, Class-Modulated convolution (CMconv), to allow the training data to be shared among different architectures, so as to balance the training data across classes. The proposed method leverages a Markov Decision Process (MDP) in the search algorithm, and learns the sampling policy for NAS. Comprehensive experiments demonstrate the class-aware NAS can outperform class-agnostic NAS.

- The paper is well written with sufficient figures and plots. 
- The proposed idea is straightforward and convincing.
- Rich experiments and analysis are conducted. Implementation details are clearly described in the appendix.
- I like the figures architecture searched specifically for different classes.

However, I still have some concerns:

- From my point of view, the proposed CMconv has exactly the same architecture as the one in Karras et al., instead of using class embedding as input. Please clarify the difference and clearly point out in the paper.

- Quantitative results in Table 2 seem not promising. The proposed method is not compatible with the SOTA. Although the paper claims that the proposed idea can be applied to the existing methods and performs better, there’s no evidence showing that.

- It’d be better to also list infra-FIDs of one existing unconditioned GAN method in the Table 1, for the lower bound / baseline of the experiments.

- For fair comparison, it’s better to have a table with quantitative results of IS/FID on CIFAR-10, listing the existing methods with conditioned/unconditioned, such as AutoGAN, style GAN etc. 

Overall, the proposed method is interesting, NAS by MDP with class-aware functionality, which can ideally outperform the class-agnostic based method. The experiments are comprehensive, with strong ablation study and analysis. However, it still requires some clarification and convincing experiments to demonstrate the performance. 

I'm willing to rate higher if the concerns are addressed.
review_title: Class-awareness mechanism involved in the NAS approach

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:  
This paper proposes to use neural architecture search (NAS) to automatically discover useful conditional generative adversarial network (cGAN) architectures. Specifically, this work aims to find a dedicated architecture for each class.  

Strengths:  
-Paper is well written.  
-Demonstrates that optimizing architectures for each class yields some improvement over using a single architecture for all classes.  
-Architectures learned by the NAS reveal insights about how to use existing building blocks, such as where best to place feature modulation layers in the network.  

Weaknesses:  
-No random baseline (see [1]).  
-Given how close NAS-cGAN and NAS-caGAN are in terms of performance, confidence intervals should be reported to confirm that improvement is significant.  
-Majority of improvement comes from fine-tuning on each class individually. It is unclear how much of this improvement is simply due to additional capacity.  
-Complexity of the model appears to be disproportionate to the improvement in performance (lots of implementation effort for a somewhat small gain in performance).  

Recommendation and Justification:  
While I think this paper is well written, after reading it I am not convinced of the usefulness of the core idea, which is that generator architectures should be class-aware. It is never explained why it might be desirable for each class to have a distinct generator network, and I cannot think of any reason why this may be the case aside from increased model capacity. For this reason I think that this paper is currently marginally below the acceptance threshold, but look forward to the author's explanation.  

Clarifying Questions:  
-How is calibration performed exactly? Is this procedure the same for NAS-cGAN and NAS-caGAN? Is a separate copy of weights fine-tuned for each class?  If a new copy of weights needs to be created for each class, proper comparison would be to a non-NAS model with up to n_classes times more model capacity than the base model.  

[1] Li, Liam, and Ameet Talwalkar. "Random search and reproducibility for neural architecture search." Uncertainty in Artificial Intelligence. PMLR, 2020.  
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################
 
Summary:

This paper proposes a framework NAS-caGAN that adopts RL-based NAS to search the optimal class-aware generator architecture by directly optimizing the Inception Score (IS) using the  REINFORCE algorithm, and leverages the mixed-architecture optimization to mitigate the training data sparsity of each category. The authors design a Class-Modulated Convolution to allow for the weight-sharing among different searched architectures. The proposed NAS-caGAN outperforms the model that employs searched class-agnostic architecture on CIFAR 10 and achieves better results compared with cproj (Miyato & Koyama, 2018) on CIFAR 100. 
 
##########################################################################
 
Reasons for score: 

Overall, I vote for rejecting, but I am happy to modify the score if the authors could provide further details about my concerns. My major concerns are about the baseline choice, computational costs, and the evidence to support the method’s utility (see cons below).  Hopefully the authors can address my concern in the rebuttal period. 
 
##########################################################################
 
Pros: 
1. The idea of using an RL-based NAS to search for the optimal architecture of different categories is quite interesting and empirically demonstrates its superior effect when compared with the searched identical generator architecture regardless of the category class.
2. Overall, the paper is well written and technically sound. 
3. Good ablation test that highlights the utility of the class-projection (cproj) discriminator over a standard GAN’s discriminator.
4. Good illustration of the proportion of different operators (i.e., RConv and CMConv) in each layer of the class-aware architecture.
 
##########################################################################
 
Cons: 
1. The key concern about the paper is the lack of experiments to validate the utility of the proposed method compared with the previous work. Despite the paper asserting that the usefulness of searching for distinct architecture for different categories, the paper only compares the **searched** class-agnostic model (NAS-cGAN) with the proposed class-aware one (NAS-caGAN), and ignores the comparison with the literature on CIFAR 10, such as (Brock et al., 2018; Kavalerov et al., 2019; Zhao et al., 2020) .
2.  Another concern is the baseline presented. 
(1) Most recent works such as (Kavalerov et al., 2019; Zhao et al., 2020) selected BigGAN (Brock et al., 2018) as their baseline. Considering the limited results (not achieving the state-of-the-art on neither of the presented metrics), a deeper analysis and performance comparison of the proposed framework would have been helpful to argue for its effectiveness. 
(2) It is interesting to inject CMConv operation at the first layer of BigGAN, which achieves a better result than other operator settings. Why not use BigGAN as the baseline?
3. Could the authors provide the details about computational and **time** costs of the proposed method? How much time/resources would this method take to search?  
4. In the Sec.4.1, the paper claims that this method could benefit other works. Considering the lack of convincing experiments to support this argument, it is doubtful about its correctness.
5. For the “moving average”, it would be better to provide more details about it. The paper claims the usage of it for training stability, but never mentions the details about it, which seems not very clear to me.
6. In the Sec.2, it would have been nice to supplement the details compared with the recent literature of GANs with NAS.
 
#########################################################################
 
Minor comments: 
1. Table 3:  The reported evaluation score is not mentioned, which might be FID.
2. The “moving average” has not been well addressed in the main paper. 
3. Section 3.1: it would be easier to follow if the authors could paraphrase the last two paragraphs. 

review_title: An interesting idea but may lack of some supportive experiments (see cons).

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an interesting method that adopts NAS to search multiple class-aware generator architectures for cGAN instead of class-agnostic type. A search space containing both normal and class-modulated convolutions are introduced to simplify the process of re-training. Besides, this paper design a mixed-architecture optimization to specifically address the computational burden issue under the setting of a multi-net search. The search results also give some insights about constructing cGAN models.


Strengths:
- The perspective of adopting the NAS method to explore the class-aware generator architectures for conditional GANs is relatively novel and interesting, although there are some related works about searching unconditional GANs. 
-The proposed flexibility and safety search space is effective to address the categories grow issue. 
- The developed mixed-architecture optimization is a clever way to improve the efficiency of the search and re-training process.
- The authors conduct extensive experiments and give some interesting insight/discussion about the results.

Weaknesses:
- There are no innovative approaches toward neural architecture search are proposed, and this work only focuses on how to bring existing RL-based NAS methods to cGANs while overcoming some issues.
- The motivation for using distinct architecture to generate images for each class instead of using one architecture is unclear. Most of the existing cGANs in noise-to-image and image-to-image translation settings employ CBN or AdaIN to embed conditional information to a unified generator. 
- The idea is similar to the dynamic routing/inference networks such as Blockdrop [1], and it needs a related discussion about the difference.
- The experiments are only conducted on low-resolution datasets. In my view, the GANs search algorithm needs to be verified on high-resolution datasets, instead of still continuing to achieve marginal performance improvements on small datasets.

Wu, Zuxuan, et al. "Blockdrop: Dynamic inference paths in residual networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

review_title: Overall a good work, but the motivation is not quite convincing, and the novelty is somewhat limited. Experiments on higher-resolution datasets are expected.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:
This paper designs class-aware generators (increasing flexibility) for cGAN by RL-based neural architecture search (NAS) algorithm. GAN models with better flexibility are promising to yield better performances.

Class-Modulated convolution (CMconv) is designed to increase model flexibility while enabling data sharing among classes, increasing the efficiency of training data.

Mixed-architecture optimization is presented to ease the training procedure of multiple class-aware generators.

Cons:

The proposed Class-Modulated Conv also inserts the class embedding to a common convolution, which is similar to a regular BN. The architectures for different categories are still weight-sharing, which is quite a common approach in NAS. Thus, this work may be treated as applying NAS to cGAN.

For searching network for GANs, the main challenge lies in how to provide stable and efficient supervision as a reward. Note that during training, the generator (G) and discriminator (D) play as rivals. Searching for the architectures of G and choosing IS as a reward only helps the G compete against D. I wonder whether it is the optimal choice, so the authors need to consider the settings in [1], i.e., updating the architectures for both G and D.

Many NAS algorithms for GAN models are relevant to this work, but none of them is evaluated against the proposed method in the experiments. I think the authors should add competing results from AdversarialNAS [1] and AutoGAN. Specifically, the Intra FIDs on CIFAR10, FID, and IS scores on CIFAR100 of the two methods should be reported.

Neither time complexity nor space complexity is reported. Considering that the resolution of outputs is only 32$\times$32, I wonder whether the proposed method is prohibitively expensive to be applied to practical applications that require higher resolution. Note that the resolution for most generation tasks is at least 256$\times$256 or 128$\times$128. I suggest the authors report the training and testing times, as well as the consumed GPU resources.

 
[1] Gao, Chen, Yunpeng Chen, Si Liu, Zhenxiong Tan, and Shuicheng Yan. "Adversarialnas: Adversarial neural architecture search for gans." CVPR, pp. 5680-5689. 2020.

[2] Gong, Xinyu, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. "Autogan: Neural architecture search for generative adversarial networks." ICCV, pp. 3224-3234. 2019.

review_title: This paper applies Neural Architecture Search to Conditional GAN. The Class-Modulated Conv is proposed to condition the search space.

Meta-review: This paper investigates the use of class-conditional architectures in GANs. It achieves this by employing neural architecture search (NAS) on top of reinforcement learning. Their main contribution is a “flexible and safe” search space; experiments are carried out on CIFAR-10 and -100. Standard performance results are augmented by diagnostic studies.

This paper received a total of five reviews which, remarkably, yielded the same assessment: the paper had merits but was marginally below the acceptance threshold. In general, the reviewers thought the idea was interesting and straightforward, experiments extensive and the paper was clearly written. The primary concerns brought up were novelty (i.e. just cGAN + NAS?, R1 and R2), minimal performance gain (R4, R5), unclear motivation (R2), lack of comparisons --- including to other NAS for GAN methods --- (R1,R3), limited to low-resolution datasets (R2,R3), no reporting of time or space complexity (R1,R3), unclear where improvement comes from --- no control for capacity --- (R5). 

On the point about comparing to NAS + GAN works, the authors responded, stating that the NAS + GAN methods brought up by the reviewer were unconditional GAN methods and pointed out that they made unconditional GAN comparisons in the Appendix.

The authors also emphasized to multiple reviewers that the point of the paper was not to improve NAS. Interestingly, they also made a comment to R5 that the point of the paper was not to improve performance of cGANs, but to improve understanding of them.	

The reviewers are unanimous in that this paper falls just below the bar for the reasons outlined above. Following the discussion phase, I see no reason to overturn their recommendation. I hope that the authors can use the feedback from these reviews to improve this paper and re-submit it.
Decision: Reject