Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
This paper tries to solve the data imbalance problem in conditional GAN by adding a classification loss as the constraint. This constraint can be seen as weighted softmax and the weight is the smoothed number of classes showed before. 

This paper is well written and structured. The idea is intuitive. The authors applied the methods in long-tail classification to GAN. In the experiment part, the authors first did the image generation from long-tailed distribution on CIFAR10 and LSUN, the apply this technique to data-free universal adversarial perturbation.  The results are better than the ones without the proposed constraints.

My main concern is that the dataset used in the first part is too simple to reflect the strength of the algorithm. Since the method is very intuitive, more experiments on datasets with larger pixel number and more categories are more convincing. 

Another concern is the baseline. Since current state-of-art conditional GAN can reach lower FID (compare with FID score in the last column of Table 1), it might be better to compare with those methods. And there exist other intuitive baselines like data-augmentation and resampling.  is it fair to compare your methods with these?

 
review_title: The idea is intuitive and hope to have more experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: Paper summary:

The paper proposes a regularizer to force an unconditional GAN generator to produce samples that follow a uniform class distribution. To provide feedback to the generator about the class distribution over the generated images, the proposed method utilizes a pretrained classifier on the same (imbalanced) training dataset. Motivated by the exponential forgetting of earlier tasks in neural networks [1], the regularization term encourages the generator to increase the proportion of samples of an infrequent class after a certain number of iterations and vice versa. Empirical studies are performed to show the effectiveness of the regularization: 1) the paper shows that the proposed method enables generating samples with a uniform class distribution with a GAN trained on a dataset with a long-tailed class distribution and (2) that the method benefits in generating universal adversarial perturbations (UAPs) in the data-free scenario. 

Pros:

1.	The paper studies an important and challenging problem of training GANs on imbalanced dataset.
2.	The proposed regularization term is novel, the derivation of the regularization term is well explained.

Cons:

1. Image Generation

-	It is not clear whether it’s necessary to obtain a set of balanced samples in such a complicated way. Given a pretrained classifier as used in the proposed method, we could simply use the classifier to select samples after training a standard unconditional GAN. This simple baseline experiment is missing in the paper.

-	The experimental setup in Section 4.1 might be unfair for the unconditional GAN baselines. If the model is trained on an imbalanced dataset and tested on a balanced dataset, then the comparison between the proposed method and unconditional GANs is not fair. FID for the latter might be worse simply because the training and test distributions are different, whereas the proposed method is tailored for this special purpose. 

-	The paper highlights that the method can generate images with uniform class distribution even in the unconditional case, where no labels are given to the generator. This would be useful if the classifier training is decoupled from the training data of the generator (this is not the case in the presented experiments, as the classifier uses the same training data as the generator with GT class labels). E.g. the classifier is trained on one dataset and then transferred for GAN training on another similar but unlabelled dataset, producing sharp images, or if only part of the training data was labelled, thus reducing the need for labels. Such experiments would help to support the claims in the paper. After all, the method is quite interesting, since in contrast to a conventional conditional GAN, the discriminator is not provided with the class identity of the generated image directly. 

- The method and evaluation heavily relies on the quality of the pre-trained classifier. In the presented experimental setup, the classifier is trained on the same training set as the GAN model. So in case of the highly imbalanced training set, it's not clear how well the classifier can recognize the imbalanced classes. Thus the proposed model might suffer from the same problem as the unconditional GAN if the classifier has troubles recognizing imbalanced classes and is biased towards well represented classes.

- The problem posed in the paper is that unconditional GANs do not sample images from the present classes uniformly, but are biased by the class distribution in the dataset. This makes sense in the unconditional case, but Figure 2 shows that this also happens in the conditional case for ACGAN. Some further explanation for why that is would be helpful. Figure 2 is not discussed in the text, but it should be. 

-	The produced samples are uniformly classified into different categories by the classifier. It would be good to provide a figure with generated images and the corresponding predicted labels, to see if the prediction and actual content match as well as in the case of a conditional GAN, like cGAN.

-	The proposed method is also trained with a big batch size of 256, which is very helpful for covering all classes. It would be more useful to see if the method also works well with small batch sizes of 16 or 32, which are common for high resolution GAN image synthesis.

-	Both the conditional GANs and the proposed regularizer use the same labelled data.

-	The paper says that in the highly imbalanced case cGAN sufferes from the training instability. In this case how is the batch formed for cGAN training? Do you balance the batch in terms of classes (as the batch size of 256 is quite large)? If not, it would be interesting to see how cGAN performs with the balanced batch.

2. Universal Adversarial Perturbations

-	The experimental results in Section 4.2 does not look convincing. The listed methods have different degrees of available information (either overlaps with the target dataset or a pretrained classifier on the target dataset). Moreover, for the proposed method, the requirement of a pretrained classifier on the target dataset is a strong limitation.

-	The paper claims that the method can help generate UAPs in the absence of the target dataset for which a classifier shall be fooled. The target dataset is ImageNet. The dataset on which the auxiliary classifier for the proposed regularization loss is trained is also ImageNet. Hence, it seems the UAPs could have been learned from ImageNet directly. To avoid confusion, sentences like the following need more elaboration: "We also find that our data free results are at par with the recently published method (Zhang et al., 2020) which uses ImageNet training data."

-	The following sentence needs more explanation: "Our approach achieves diversity through sampling from multiple checkpoints, as in each cycle the regularizer encourages the GAN to focus on different poorly represented classes". If checkpoints from different training steps are used, it seems that the target distribution is not really captured uniformly, contrary to the aim of the paper. Hence, it would be good to add some explanation here to prevent misunderstandings.

-	Table 2 is not discussed in the text.

Minor comments:

-  The height-width ratio of Figure 1b should be rectified.
-  The discussion is not well written. Especially the message of the 2nd bullet point needs more clarifications.


Review summary:

The paper shows that the classification and image generation can be decoupled for GANs. This makes the setup semi-supervised, not unconditional (as claimed in the paper). While this is interesting, it would be good for the paper to show an application where this is actually useful, because in all provided examples the ground truth labels of the training set are present and used for training the classifier, but are just not used directly to train the discriminator. For example, when it comes to UAPs it is not clear to what extent the proposed approach is data-free if ImageNet is used for training the regularizer as well as the classifier to be fooled. Thus, the main weaknesses of the paper in my opinion are the significance of the presented results, unfairness in the experimental setup, and the clarity of presentation. 


Post-rebuttal feedback:

Thanks to the authors for the provided extra experiments and clarifications. I feel that my concerns have been partially addressed, thus raising my score to 5. I still think that the proposed method is limited by the classifier, it's ability to capture a long-tailed distribution, which is not so easy to get when trained on imbalanced dataset. This significantly limits the applications of the proposed approach in real-life scenarios. The paper has also experimented only on artificially created imbalanced datasets, which contain small number of classes with the model being trained with the batch size higher than the number of classes. It would be beneficial to see how the model would perform in more realistic setup when the number of classes is significantly bigger than the batch size (e.g. iNaturalist or even ImageNet), to support more the claims of the paper.
review_title: Questionable significance of the presented results

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper focuses on the problem that GANs' poor performance in the imbalanced dataset and presents the class balancing regularizer for training GANs, encouraging the GAN to pay more attention to underrepresented classes. 
They induced the class distribution information using a pre-trained classifier, and the regularize utilizes the class distribution to penalize excessive generation of samples from the majority classes, thus enforcing the GAN to generate samples from minority classes. 

Pros:
+ The motivation is clear.
+ It seems to cite the relevant literature (that I know of) and compare it to reasonably established attacks and defenses.
+ Simple/directly applicable approach that seems to work experimentally, but

Cons:
- The method part is not easy to follow. My understanding is the effective class frequency is the cumulative number of generated samples for each class (with a discount factor) and normalizing it yields the distribution of the generated samples. But I didn't get the relationship between this distribution and the following regularizer. Could you comment on that?
- As far as I understand, the regularizer encourages all the classes to be balanced within each batch by maximizing the entropy. I am a little bit concerned about this setting, what if we use a small batch size so that the class distribution can be imbalanced within one batch.
- Only LSUN subset and CIFAR10, which include 5 and 10 classes respectively. I am wondering about the performance on large-scale imbalanced datasets like iNaturalist.


----
Updated:
Thanks to the authors for the provided extra experiments and clarifications. Some of my concerns (e.g., how the batch size affected the performance) have been diminished, but I do agree with other reviewers that more baselines should be included. 
review_title: The method part is not clearly written.

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Overview**: The paper presents a simple regularizer term that aims to force a GAN to generate samples following a uniform distribution over different classes. The regularizer depends on a classifier that works well on an imbalanced or long-tailed dataset. The paper presents experiments on CIFAR-10 and LSUN that were synthetically long-tailed or imbalanced. The results show that the proposed term generates samples that follow a more uniform distribution over classes.

*Pros*:
- Interesting idea as it can help a generative algorithm to remove an imbalance from a dataset. 
- The proposed regularizer is simple but depends on a classifier (see below for more details).

*Cons*:
- The regularization term depends on a classifier that works well already on the imbalanced dataset. Getting a classifier to work on long-tailed datasets is not an easy task and people are still investigating the development of techniques to learn from imbalanced datasets (see for example I).  From a practical point of view, this is a hard requirement that can reduce the chances of adoption.

- Proposed loss may have conflicting terms. The final loss composed of the relativistic loss and the regularizer may be conflicting. According to the text (below Eq. 3), this loss follows the training distribution which in the context of the paper is long-tailed. However, the proposed regularizer penalizes the GAN to generate samples following a long-tailed distribution. Aren't these two terms then conflicting? If so, can this conflict impact the convergence of the network?

- Insufficient experiments. While the experiments show good results on two small and synthetically long-tailed datasets, it is unclear if this method can work on naturally long-tailed datasets (e.g., iNaturalist). Unfortunately, the CIFAR-10 and LSUN datasets have a small set of classes in them. How does this method work on naturally long-tailed (e.g., iNaturalist) and/or large-scale datasets with a larger set of classes (e.g., ImageNet-LT)? Also, how do the generated images look like? Does this method still preserve a good perceptual image?

- Lack of clear impact on applications. After reading the introduction, I did not have a clear application where this component can be crucial to either enable a new application or solve a bottleneck. The discussion section briefly mentions a few applications. However, I think the paper would've been stronger if it showed experiments using the proposed approach and showing its impact on a clear application.

References:
I. Liu et al. Large-Scale Long-Tailed Recognition in an Open World. CVPR 2019.

Minor comments:
1. The contribution list of the Introduction section uses terms that have not been defined, i.e., FID and UAP.
2. If using latex, please use \min, \max, \log to properly display the operators.

----------------------------------------------------
Post Rebuttal Update

While I think the idea is interesting, I still think the proposed loss is not consistent as I still think the two terms in the loss collide with each other, its practical value is limited mainly because making a GAN to work on various datasets is a challenging task, and that the experiments now raised more questions than answers. For these reasons I still lean towards rejection as I believe the paper can benefit from a revision.
review_title: Interesting Simple Idea but not Convinced about Motivation and Results

Meta-review: The authors have provided very detailed responses and added additional experimental results, which have helped address some of the referees' concerns. However, since the modification made to a vanilla GAN algorithm is relatively small, the reviewers are hoping to see the experiments on more appropriate real-world datasets (not artificially created imbalanced datasets with relatively few classes), more/stronger baselines, and rigorous theoretical/empirical analysis of the method's sensitivity to the quality of the pre-trained classifier. The paper is not ready for publication without these improvements. 
Decision: Reject