Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors of the paper propose a unified view of conditional generative adversarial networks (cGANs). To this end, they analyze the log of joint probability distribution p(x, y) through two different perspective of views; via a conditional discriminator using Eq.(2), and via an unconditional discriminator and classifier using Eq.(3). Inspired by a multitask learning, they suggest Energy-based Conditional Generative Adversarial Networks (ECGAN) where two approaches are combined together to approximate the joint probability distribution. Under this framework, previous cGANs such as ProjGAN, ACGAN and ContraGAN can be closely explained using the variants of ECGAN with the right choice of hyperparameters, namely, ECGAN-0, ECGAN-C and ECGAN-E, respectively. 
Their experiments on CIFAR-10 and Tiny ImageNet show that these ECGAN variants outperform ProjGAN, ACGAN and ContraGAN in most cases of experiments. Furthermore, ECGAN-UC outperforms these variants of ECGAN as well as previous cGANs with various backbone structures such as DCGAN, ResGAN, and BigGAN. Through this result, the authors conclude that adding a classifier helps improving cGANs with the help of the unconditional discriminator as opposed to the previous findings in Shu et al. [42] and ProjGAN [34].	
main_review: Originality:

Previous cGANs have taken one of approaches explained in section 2.2 and 2.3 in a broad perspective. By combining them, this paper indeed provides a unified picture of cGANs. Although the concurrent work [11] is somewhat similar in that they both try to solve Eq.(5), the tasks they are trying to solve and the approaches they are taking are quite different as mentioned in the related work.

Quality:
- Line 249 ~ 250 - The authors mentioned that the entropy estimation lower bound provided by the contrastive loss is too loose to benefit the training. But, doesn’t ECGAN-UC correspond to using 0 as a lower bound for entropy (1st approach mentioned to estimate entropy) which is a looser bound for entropy? 
- Line 247 ~ 248 - The authors claimed that the unconditional GAN loss is a crucial component to bridge classifiers and discriminators in cGANs according to the performance results provided in Table2. Could authors provide any idea why it is the case other than the empirical evidence?
- Their experiments are somewhat limited as they did not provide the experiment results on ImageNet which were used by all of ProjGAN, ACGAN and ContraGAN.

Clarity:
- Line 101 - It would be better to specify a section of the book or provides a proof in the appendix as done in appendix A.1 of [11].  
- Line 141 - Could authors elaborate why Eq.(10) is an empirical estimate of a proper lower bound of an entropy?
- Line 164 ~ 165 - It would be better if the authors can provide experiment results that compare the performance with the original wassertein loss versus hinge loss since it is hard to capture how much improvement is originated from using the hinge loss.
- Line 197 ~ 198 - The authors said the main difference between ACGAN and ECGAN-C is that ECGAN-C uses a conditional discriminator. But, if my understanding on the equations in section 3.2 is correct, another difference is that ACGAN has additional cross entropy loss for the generated images. Do authors think it is negligible? Also, wouldn’t ECGAN-C closer to ACGAN if unconditional discriminator is used instead of conditional discriminator? According to the claim in line 247 ~ 248 that unconditional GAN loss is crucial component to bridge classifiers and discriminators in cGANs, I am not sure why ECGAN-C is set in that way.
- Typos:
    * It seems the denominator part of Eq.(10) is missing an indicator function of (k != I) according to the Eq.(8) in ContraGAN.
    * Line 161 - increase -> increases, decrease -> decreases
    * Eq.(14) - Third term in the middle h(x) -> g(x)
    * Table 4 - bolding is missing for some cells. 
    * Citations [11] and [12] are mixed up in the related work section.

Significance:
- Although there are some remaining questions as specified above and experiments are somewhat limited, this work can benefit the community by providing a unified framework for cGANs. 
limitations_and_societal_impact: Some limitations that the authors did not recognize are provided above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors target to leverage classifiers in a principled manner and unify them in cGAN, by using the decomposition of joint probability distribution and a classic energy model to parameterize the distribution. Experiments show new state-of-the-art generation quality in terms of FID and IS, over several datasets (CIFAR-10, Tiny ImageNet) on several GAN implementations (DCGAN, ResGAN, BigGAN) and against several baselines (ACGAN, ProjGAN, ContraGAN).
main_review: ********************************************************************
Strengths
********************************************************************
+ Clear exposition: good writing and good result demos.
+ The research problem is clearly defined.
+ Reproducible.
+ Experiments are dense and results are informative to set the new state-of-the-art.

********************************************************************
Weaknesses
********************************************************************

- Experiments can be more convincing if not only on toy datasets like CIFAR-10 or Tiny ImageNet. Repeat the quantitative comparisons on the entire ImageNet dataset with the full resolution.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduced a general framework for conditional image generation. And existing cGAN methods such as ACGAN, ProjGAN and ContraGAN are inclued in the proposed algorithm. Experimental results in Cifar and Tiny ImageNet verified the effectiveness of different components of the proposed method.
main_review: This workd presented a detailed discussion among the differences on existing condiontal image generation methods. The unified view of cGAN is interesting. However, the authors did not prpose new loss functions for cGANs. 


Thanks for the author feedback. Some issues about the experimental evaluation have been addressed welll. However, I still concern about the originality of this work which is also indicated by Reviewer gvcB. In addition, the new results on ImageNet are also confusing and not satisfactory. According to the table below, your reinplementations of ACGAN, ProjGAN and ContraGAN are better. Then the comparisons with BigGAN, ContraGAN based on StudioGAN are not convincing.
limitations_and_societal_impact: 1. The experimental results are not convicing. The authors should empoly datasets with higher resolution such as $128\times128$, $256\times256$. The qualitative results in Fig. 1 and Fig. 2 are very blurry.
2. It seems that the reported FID results of existing methods are low compared with the PyTorch-StudioGAN. Please give more explanation.
3. The performance of different variants in Tab. 2 is not consistent. Could the classification loss and entropy estimation loss help the conditional image generation?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This submission proposes to analyze the most popular variations of conditional GANs (ACGAN, ProjGAN, ContraGAN) under a unified, energy-based, formulation (ECGAN).

Specifically, ECGAN is composed of terms derived from two different decompositions of the joint probability p(x,y).
The paper then links each term to one of the popular cGAN approaches.

ECGAN is evaluated against the traditional models and an extensive ablation study tests the impact of each component of the proposed model.

main_review: Originality:
The paper builds very heavily on existing work. The ECGAN formulations are only marginally different from previous works.

Quality:
The submission seems technically sound, supported by adequate theoretical framework and experiments.
It is however difficult to draw conclusive results on only CIFAR-10 and Tiny ImageNet.

Clarity: 
The paper is clear and well organized. 
The association between the different cGANs and their corresponding ECGAN variants could be debatable, however (see questions below).

Significance: 
The proposed formulation provides a unified view of the different losses terms used in conditional GANs that allows for a valuable principled evaluation.
Moreover, considering that ECGAN-UC has very encouraging reported performances and that its implementation is only a slight departure from previous cGAN's, it could become a staple for conditional GANs if future works confirm its good performances.

Overall, I believe the paper is interesting to read and of good quality. 
I still have some questions and remarks that I hope can be clarified:

- Since ProjGAN sums both an unconditional and a conditional output, wouldn't it make more sense to compare it to ECGAN-U instead of ECGAN-0? Especially since the used Hinge Loss implementation, described in appendix A, sums the conditional and unconditional terms before the Hinge operation, just like in ProjGAN.
I would appreciate it if the authors could comment and discuss the differences of model and performances between ProjGAN and ECGAN-U.

- Similarly, ACGAN is compared to ECGAN-C in section 3.2. But in section 2.2, ACGAN is already viewed as an unnamed variant of ECGAN (without the ECGAN-0 term and alpha=1 instead).
Could the authors comment on why ACGAN is not directly considered a specific variant of ECGAN?
Considering both previous points, my opinion is that the comparisons in section 3 seem forced and actually make the paper less clear.

- Additional experiments on diverse datasets would strengthen the results, even if performed only with ECGAN-UC and a baseline.


-------------------------------------
Post-rebuttal:
I'd like to thank the authors for their detailled responses and for providing additional results.
After reading the different reviews and responses, I believe the proposed framework is promising and the comparison between the different variants can provide interesting insights for GANs.
However, I also believe the paper needs to be very clear about exactly how the classical cGAN fit or don't fit into the ECGAN formulation.
Without the ability to assess a revised version, I remain unconvinced that small modifications would be sufficient in that regard.

I therefore keep my previous rating (6: Marginally above the acceptance threshold).
limitations_and_societal_impact: Limitations and negative impacts are adequately discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors discuss that existing methods on including classifiers in a cGAN biases the generator in generating easy to classify images. Therefore, they propose a way to include classifiers in a cGAN to improve its performance in a principled manner. To do so, they decompose the joint probability distribution by the Bayes rule that results in linking classifiers to conditional discriminators. The proposed formulation shows that a joint generator model can be trained from two directions: a conditional discriminator and an un-conditional discriminator with a classifier.
They combine the formulation of these two routes and propose a new method called Energy-based cGAN (ECGAN). ECGAN shows how to use a classifier for cGANs, and it explains other variants of cGANs such as ContraGAN, ACGAN, and ProjGAN. They empirically show that ECGAN outperforms existing cGANs by achieving higher FID (and similar ones) score on two sets of datasets (CIFAR10, Tiny ImageNet).

main_review: Positives

•	The paper's motivation and the overall idea are interesting. 

•	Principled way to use classifiers in conditional GAN. The authors present a new view to explore cGANs and use classifiers in a way that is beneficial to cGANs. 

•	The presented framework covers and explains other variants of cGANs (ContraGAN, ProjGAN, ACGAN).

•	Experimental results confirm previous results such as adding a classifier directly to ProjGAN does not result in improvements. 

•	Based on the experiments, the ECGAN method achieves higher scores compared to the existing methods. 

•	The experimental findings are consistent across different GAN architectures. 

Negatives:

•	The additional of the cross entropy loss to equation (11) appears adhoc to cover ACGAN as one of the variants of ECGAN. 

•	The authors may add more explanation on the general architecture of the method. Are there two Discriminators with a 
 classifier and a generator? There is some confusion regarding whether the classifier and the conditional discriminator are the same neural network or separate? 

•	The authors explain how they link conditional discriminators and classifiers. It is, however, unclear why the loss components are helping? what kinds of behaviors are promoted by the loss components?

•	It would be beneficial if the authors also present training stability analysis (similar to figure 3 of ContraGAN [1]). This analysis can reveal how the formulation and losses are helpful.

•	It would be great if the authors can visualize and compare the diversity of generated images across different existing cGANs (ContraGAN, ProjGAN, ACCGAN).

•	What is the computational cost of this method compared to the ContraGAN and ProjGAN?

•	Since the results between the various approaches are not significantly different for one of the 2 datasets used (CIFAR-10), the authors could experiment with more challenging datasets (e.g., ImageNet) to demonstrate that their findings are consistent.


[1] Kang, Minguk, and Jaesik Park. "Contragan: Contrastive learning for conditional image generation." arXiv preprint arXiv:2006.12681 (2020).


limitations_and_societal_impact: Limitations and societal impact are adequately discussed in the paper.
review_title: null

Meta-review: This paper proposes a framework for thinking about several existing class-conditional GAN approaches in a unified way, which also leads to a new suggested algorithm. The algorithm outperforms existing approaches on TinyImageNet, and some initial results are shown in the rebuttal that it also does so on ImageNet. I think the framework is sensible and, from what we can tell, the new algorithm is probably an improvement over existing ones. Please ensure that you address the various concerns raised by reviewers, as well as adding the new results, in your final version.
Decision: Accept (Poster)