Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: #########################################

Summary:

The paper focuses on conditional image generation with continuous label. Current methods utilize categorical conditions, which easily suffers form two problems: (1) the inferior performance when given few images for special category (2) failure of using continue label. In this paper, authors reformulate the class-conditional GANs, and  provide two new objectives (hard vicinal discriminator loss and soft vicinal discriminator loss ) and one generator loss. Both the discriminator loss and the generator loss is extended based on the vicinal risk minimization. Authors design interesting experiments to evaluate the proposed method.

#########################################

Pros:

+The paper explores new problem for conditional GANs. Specially the continues label never be studied, which is interesting for me.

+Authors leverage vicinal risk minimization to replace  current empirical generator loss, which is new viewpoint to investigate the conditional image generation.

+The first experiment is interesting, which is suitable to support the proposed method.

+The paper has good motivation, and is easy to follow.
   
#########################################

I have a few concerns which is as followiing:

1.  In page 2, authors mention '(1) our experiments in Section 4 show that this approach often makes cGANs collapse;' what is reason why current methods fail? Is it due to the imbalance of data for each category? or Is it normal issue of training GANs and cGANs?

2. I really like 'circular 2-d gaussians' experiment, which is smartly designed the proposed method. For the proposed method (E.q. 11 for generator), it has already  seen  the data which is corresponding to  test label (angle), since the real sample (Figure 1(a)) is localized in any angle and the used label (of the proposed method) for training is noised, which means the noised label is the test label.  Could authors try the real sample like one of Figure (b) which the real sample is not continues.  

3.  I am wondering the proposed method could be used for fine-grand image generation. For example, the bird dataset[1] has 555 category,and some categories are close.

4. In paragraph: How is (P2) solved?. The new method to map the label seems similar to the label embedding. For example, the label embedding of BigGAN is also be updated. But here authors map the label by FC layers, which is corresponding to the learned label embedding. And the combination  of both the image embedding and the class embedding is similar to the one used in SNGAN and BigGAN.
    

[1] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, pages 595–604, 2015. 


review_title: CcGAN: Continuous Conditional Generative Adversarial Networks for Image Generation

final_score: rating: 7: Good paper, accept
review_content: *** Summary:

This paper proposes a novel continuous conditional GAN which takes continuous scalars (named regression labels) as conditions. There are two problems for continuous conditional GAN: (P1) cGANs with discrete labels are trained to minimize the empirical loss, but this fails for continuous conditions, because there might be few or even zero samples for many labels values. (P2) For continuous labels, the label cannot be embedded by one-hot encoding like discrete labels. The authors propose two solutions for the problems respectively: (S1) The authors give two types of estimate (HVE and SVE) of the joint probability used for calculating the loss, and further formulate new loss functions HVDL and SVDL, based on the estimated joint distribution. (S2) The authors propose new methods of encoding the label input: element-wise addition for the generator, and linear embedding layer for the discriminator. The authors further derive and prove the error bounds for a discriminator trained with HVDL and SVDL. Experiments are conducted on continuous label datasets, and the authors propose a new dataset with continuous labels. Comparisons are made with traditional cGANs with discrete labels.

*** Pros:

1. This paper is the first work on continuous conditional GAN. It solves the two key problems for generalizing traditional cGANs to continuous conditions, and empirically validates the effectiveness of the proposed approach.

2. The proposed approach is theoretically sound. The generalization from traditional cGANs with discrete labels to the novel continuous label conditional GANs is reasonable. The estimations (HVE and SVE) for deriving the new loss functions are clearly explained and mathematically elegant and solid.

3. Error bounds of the discriminator trained with HVDL and SVDL losses are derived and proved. This builds a strong and solid foundation of the proposed approach.

4. Experiments are conducted on various datasets. Reasonable results for generating synthetic data and real-world images are shown and compared with cGANs with discrete labels.

*** Cons:

1. The experiments are conducted on low-resolution images with SNGAN structure. In order to test the generalization ability of the proposed approach, would it be possible to test on other state-of-the-art structures such as BigGAN? And is it able to be trained to generate higher resolution images?

2. Conditional GANs with continuous label conditions is an interesting direction that has been rarely explored before. Since most real-world datasets are with discrete labels, and even for some labels that can be discrete, it might be impossible to label it in a continuous way, and we will assign discrete labels. Can you elaborate more on the potential application scenarios and potential impacts of the continuous cGANs, besides the viewpoint and age examples used in this paper?

*** Justification for rating

I think this paper works on an interesting direction of continuous conditional GANs, which has rarely been explored before. The proposed approach to generalize cGANs to continuous label conditions and the proposed solutions to the key problems are elegant and novel. The authors not only give clear explanations but also provides mathematical error bounds and proves. So I would recommend acceptance.

*** Questions for the rebuttal

Please address the questions in the Cons section.

*** After rebuttal

Thank the authors for the detailed explanation. After reading other reviewers' comments and the author feedback, I would like to keep my rating unchanged.
review_title: A novel continuous conditional GAN

final_score: rating: 5: Marginally below acceptance threshold
review_content:  This work proposes to perform Conditional GAN with regression labels, thus to benefit the model with data-sufficient continuations generation (vs infinite distinct condition for generation).  Inspired by VRM, a CcGAN model is proposed to tackle the challenges in existing methods. (Eq.(6) ---> Eq.(10))  An error bound is also derived for the proposed new discriminator loss. 

The idea to facilitate generation with continuous conditions is interesting and insightful. the experiments are also well-designed and analyzed to support the claim.  My questions are mainly in the following aspects:

1). Comparisons on more challenging data sets are suggested.  Also, as you argue the deficiency of requiring large sample and condition size in baseline models, comparison regarding the different number of, or scare training data are better to be presented.

2) Can you please explain your superiority and connection with [1].

[1]. Zhao, Shengyu, et al. "Differentiable augmentation for data-efficient gan training." arXiv preprint arXiv:2006.10738 (2020). 
review_title: Interesting idea, but experiments needs to be strengthen and polished.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##### Summary
This paper tries to make the generative adversarial network handle continuous, scalar conditions. Specifically, in order to make it work, the author set an empirical estimate that every small perturbation to the condition y results in negligible changes to the conditional distribution.  The author shows comparative performance improvement over the basic cGAN on circular 2D Gaussians, RC-49, and UTKFace dataset.

##### Strength
First of all, this paper is well-written and easy to follow. This paper tackles one of the important topics in a generative network with appropriate assumptions and shows its relative effectiveness not only on a single dataset but also in various settings. It also provides rich details with the codes in supplementary.

##### Weakness
First of all, although the math and logic are quite easy to follow, it is a bit hard to guess, especially to the person who doesn't have a strong background in this domain, what exact behavior is enforced to the model. Also, along with the first issue, it is hard to predict this method's stability for some non-well curated/designed setting. Please check the question section for the details.

##### Question
1. As I mentioned in the weakness part, it is quite hard to imagine how this method will behave. Similar to Figure 1 of the original GAN paper (Goodfellow et al., in NeurIPS 2014), or Figure 1 of VEEGAN (Srivastava et al., in NeurIPS 2017), it would be great if the author can visualize how the distribution is matched and what kinds of behaviors are promoted/demoted by the loss.

2. As mentioned in (P1) of this paper, there are many real-world cases that there are zero or only one label exists. To show the effectiveness in such an imbalanced setting, I recommend presenting the result that (i) train people of odd ages only and tested with even-numbered ages and (ii) tested with fewer data in some of the ages (unlike the one in Figure S. VIII.5 or S.VIII.A).

3. Also, I also wonder what happened if the condition is not directly related (to see whether the label power is essential or not). To alleviate such curiosity, I recommend testing it by modifying the label on one of the experiments. For instance, you can try this method on the RC-49 generation, not with the angle but with the chair's volume.



##### Post-rebuttal 

I thank the authors for their thorough comments and detailed explanations for each question. I carefully read the whole, and it helps me understand the entire decision and the processes. However, I would like to suggest two things, regardless of its acceptance, but to make their claim more attractive to general readers: (i) re-sort out the indexes (maybe after rebuttal but before submitting your camera-ready version), and (ii) update figures of the sample with a bit more realistic one (with training the model on larger batch size, for instance). I hope this review phase would make your paper more powerful. (disclaimer: I did not check the soundness of the mathematical equations thoroughly but did check all the rest)
review_title: Interesting approach for an important topic, but ask for some clarifications

Meta-review: The submission proposes a novel conditional GAN formulation where continuous scalars (named regression labels) are fed into the GAN as a conditioning variable. Since cGANs with discrete labels are trained to minimize the empirical loss, they fail for continuous conditions, because there might be few or even zero samples for many labels values and also the label cannot be embedded by one-hot encoding like discrete labels. As a solution, the authors propose new methods of encoding the label. 

The paper received a clear accept, two weak accepts and a weak reject. As agreed by all the reviewers, the paper proposes an interesting framework to eliminate some weaknesses of GANs. The rebuttal adequately addresses the reviewer comments and hence the meta reviewer recommends acceptance. 
Decision: Accept (Poster)