Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors are focused on a particular class of Generative Adversarial Networks (GANs) , the conditional GANs (cGANs) which incorporating additional information into GANs.
More specifically, they focus on auxiliary classifier GAN with softmax cross-entropy loss (ACGANs) 
ACGANs suffer from challenging training as the number of classes in the dataset increases and also from generating easily classifiable samples without considering diversity.
The authors identify the source of the first problem, namely they show that gradient exploding in the classifier can cause an undesirable early training collapse, and resolve it via projecting input vectors onto a unit hypersphere. 
They also propose Data-to-Data Cross-Entropy loss (D2D-CE) to exploit fruitful relational information in the class-labeled dataset. 
To the end, they propose the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN).
They also perform experimental study which show that ReACGANs achieve state-of-the-art generation performances on several datasets.
main_review: In this paper, the authors analyse why ACGAN training becomes unstable as the number of classes increases. 
Then they propose how to solve this problem : 
1) by analytically calculating gradients of the softmax cross-entropy loss used in ACGAN. By examining the exact values of analytic gradients, they show that the unboundedness of input feature vectors and poor classification performance in the early training phase can cause an undesirable gradient exploding problem;
2) having alleviated the instability, they propose the Rebooted Auxiliary Classifier Generative Adversarial Networks (ReACGAN) using the Data-to-data Cross-Entropy loss (D2D-CE). ReACGAN projects image embeddings and proxies onto a unit hypersphere and computes similarities for data-to-data and data-to-class consideration. Additionally, they introduce two threshold values for inter-class and intra-class separability. In this way, ReACGAN overcomes the training instability and can exploit supervisory signals by explicitly considering data-to-class and data-to-data relationships, and also by implicitly looking at class-to-class relationships in the same mini-batch.

The authors do indeed identify and resolve a major problem of ACGAN but their contribution to the problem is incremental in the sense that when compared with the existing state of the art methods, their performance is similar.
limitations_and_societal_impact: 
1) The hyperparameters (\tau, m_p, m_n) seem to play an important role both on the training and on the performance of the algorithm. 
The authors say that identify their values through grid search. It would be useful to see the perrformance of algorithm for different values, as well -- ie to see how robust it is wrt different values.
Also, m_p=1-m_n seems convenient but not justifiable;

2) In Table two there are some underlined values; could please explain what the underlying means? Also, in the same Table we see that none of the algorithms attains IS close to that of the real data at the Tiny-Imagenet dataset; do the authors have any explanation about that?
 
3) Given that the paper contains graphs, I would advice the authors to employ colourblind-friendly colours so that they can be accessible from colour-blind fellows;
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method to tackle the training instability issue caused by auxiliary classification loss in ACGAN. To this end, the authors propose to modify the function to estimate the class conditional probability by replacing the denominator of the softmax function with the sum of the inner product between the reference image and its negative examples. In addition, the authors introduced margin to limit the norm of the gradients. Experiments are performed on multiple well-known benchmarks such as Imagenet and compared with several previous methods.  The metric used to compare the methods are widely used ones such as FID and Inception score. Experimental results show that the proposed is superior compared to the existing methods.
main_review: Paper motivates with identifying the gradient explosion problem especially in the early training phase in the ACGAN due to the auxiliary loss and convincingly presents the idea to tackle the problem.

The paper is well written and easy to follow. All the notations, facts, and figures are clear to understand.

The experiments are extensive and the theoretical justifications seem correct. 
limitations_and_societal_impact: This paper addresses the issue with the existing GANs frameworks in order to improve the quality of image generations. Hence, I don't foresee any new negative impact that GAN  this paper can cause.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes the remedy for issues present in ACGAN formulation which causes mode collapse.  The major contributions are:
- Comparison to relevant state-of-the-art baselines has been done.
- The paper is lucidly written and the authors have done extensive experiments.  
- Code for the proposed method has been provided by the authors. Great effort has been made by authors to ensure reproducibility.

However paper claims to have rebooted ACGANs but the D2D-CE discriminator used, seems dissimilar to the ACGAN and seems more similar to contrastive loss. In my opinion, I feel there is a clear mismatch between claims and results. 

main_review: ### Clarity

This paper claims to Reboot ACGANs but in the paper the best performance is achieved by D2D-CE loss which is very similar to the contrastive loss used in contraGAN. The auxiliary classifier in D2D-CE performs inferior to the ACGAN classifier as shown in Appendix Table A9, which clearly shows that the discriminator with D2D-CE loss is definitely not an auxiliary classifier (as it has poor classification performance compared to ACGAN). I do appreciate the finding that normalization of embedding helps in improvement of ACGAN, but the final model is clearly dissimilar to ACGAN. Hence in my opinion the claim of rebooting ACGAN is a bit misleading.


### Originality: 

I find that a lot of ideas in this paper have already been explored in the ContraGAN [K1] paper

1. Idea of Normalization: The idea of normalization of embeddings such that they lie on a sphere is also used by ContraGAN.
2. Idea of Positive Negative Repulsion: The idea of introducing a positive negative based repulsion seems similar to the contrastive loss in ContraGAN.
3. Eq.5 in the paper is very similar to Eq. 7 in ContraGAN paper which is the base equation both papers build on.
4. The authors have introduced D2D-CE which seems to be inspired from hard mining based contrastive losses and afaik is novel in GANs. 
The works in appendix references [25,26,27] already have introduced the hard mining losses for classification, hence I recommend to use these losses as well and show a comparison, to make the submission more stronger.


### Quality 

There have been a lot of experiments done for verifying the effectiveness of the proposed method. I find that gains in performance for the CIFAR-10 and Tiny Imagenet overlap in the standard deviation intervals, hence the claim of state-of-the-art cannot be fully verified on these datasets. The technique seems to perform better in case of the CUB dataset.

### Significance 

I find that a lot of the ideas stated in this paper have already been explored in ContraGAN paper. Also the proposed loss is in my opinion, is more similar to contrastive loss rather than ACGAN loss. The description of the current work as Rebooting ACGAN seems misleading as the D2D-CE method is a contrastive method based on hard-mining of samples.


[K1] Kang M, Park J. Contragan: Contrastive learning for conditional image generation. arXiv preprint arXiv:2006.12681. 2020 Jun 23.

limitations_and_societal_impact: I feel that limitations haven’t been discussed appropriately in the manuscript. A line of description on limitations is definitely not enough. More discussion on additional training time due to the new loss. Also some things like how much time was required for choosing hyperparameters can also be discussed in limitations.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper tackles limitation of ACGAN.
1. ACGAN struggles when # classes is large.
2. ACGAN generates easily classifiable samples (=lacks diversity).

Th authors argue that
1. Gradient exploding in the classifier leads to collapse. Normalizing input vectors to the classifier resolve it.
2. data-to-data cross-entropy exploits relational information (?).

Performance is measured in IS, FID, PR on cifar10, cub, tiny-imagenet, and imagnet.
(Misc.) ReACGAN benefits from consistency regularization and diffaug.
main_review: Originality.
+ (+) Showing that the feature norm and the gradient (dL/dw) norm has connection to the performance is interesting.
+ (+)Combining the standard cross-entropy for classification and contrastive loss is novel (eq. 5).
+ (+)Difference between ReACGAN and ContraGAN is clearly described.
- (-) Normalizing the representations for computing inner product has already been standard in representation learning (simclr, mocov2, clip ...). They should be covered as related work.

Quality.
+ (+) Hyperparameter search is provided in the appendix. FID is mostly robust to the choices.
- (-) ACGAN's early training collapse is not rigorously defined and not shown either empirically or theoretically.
- (-) Incorrect argument: As $F(x_i)^Tw_k$ arises, p gets closer to 1, (1-p) gets closer to 0, and the norm of the gradient shrinks.
> L108 The norm of the gradient begins to explode as the inner-product value of $F(x_i)^Tw_k$ arises
- (-) The paper does not cover multi-task discriminator which has is a strong standard in recent methods [FUNIT, StarGANv2, Which training methods do actually converge]. Describing that the multi-task discriminator does not have similar phenomena will make the argument on ACGAN's instability stronger.
- (-) Reasons for using DiffAug and not using [ADA] are not described.


Clarity.
+ (+) The proposed method is clearly described enough to be implemented.
- (-) Thresholds m_p and m_n should be termed margins instead of thresholds, following [Learning shallow convolutional feature descriptors with triplet losses].
- (-) The proposed method is referred to as ReACGAN and Big+D2D-CE interchangeably. Using a single term is recommended to avoid confusion.

Significance.
+ (+) The proposed method is effective on ACGAN and ProjectionGAN variants.
+ (+) The proposed method synergies with consistency regularization.

Main reasons for my rating toward rejection come from logical weaknesses. Please refer to the <Quality> section.
limitations_and_societal_impact: They mentioned necessity of the hyperparameter search as the limitation.
In my opinion, it is not important to describe potential negative societal impact because the paper tries to improve general quality of GANs.
review_title: null

Meta-review: The authors identify that the auxiliary classifier GAN (ACGAN) fails to scale to more classes and is biased towards easily classifiable examples because ACGAN can suffer from gradient explosion when training the classifier, causing the collapse of training. They then propose to normalize the feature vectors and add a data-to-data loss function to improve the performance. The resulting model is named Rebooted ACGAN (ReACGAN), which can be easily combined with other techniques like consistency regularization for better performance. Experimental results reveal some competitive performance of ReACGAN.

The reviewers generally appreciate the clarity of the paper and the convincing argument from the claimed issue of ACGAN to the improved model. One issue that could be addressed easily is to rigorously define and describe what the authors mean by early training collapse. The major issue that was not easily addressed is on how the authors can establish originality with respect to

(1) the widely-used normalization technique in other GANs and representative learning works. The authors argue that their in-depth investigation, instead of the more ad-hoc use of other works, is an important contribution. While the contribution is indeed of some value, the authors only position this investigation within the narrow angle of the ACGAN model. It is not clear whether the contribution can also be generally used to explain the normalization technique used in other GANs. This narrow angle limits the impact of the contribution.

(2) the ContraGAN work which has a contrastive loss that shares some similarity to the proposed data-to-data loss function. While some reviewers believe that the difference between ReACGAN and ContraGAN is clearly described, and the authors argue that ReACGAN and ContraGAN losses are designed with different principles (Line 177), other reviewers point out that more efforts are needed to highlight the differences between the two loss functions, such as empirical studies on how "false negative samples" or other loss variants affect performance (like the preliminary results provided in the rebuttal). While the authors' arguments on how ReACGAN originated from ACGAN (rather than ContraGAN) is convincing, they do need more efforts to clarify whether the ReACGAN is significantly different to ContraGAN in their final form to claim for originality. Some efforts are also needed to smoothly incorporate the rebuttal paragraphs into the revised paper.

One issue that some reviewers and authors disagree on is whether ReACGAN should be "called" ReACGAN, given that the classifier nature in ACGAN was deliberately weakened for ReACGAN and the proposed loss makes the whole model look a lot more like ContraGAN. The naming issue is arguably more subjective and is not taken as the key factor behind the final decision making. Somehwo the authors are encourage to keep studying a related issue on analyzing how the classifier performance affects conditional generation performance.

The authors addressed some of the concerns with preliminary experiments, such as the comparison to multi-task discriminators to ensure whether ReACGAN is competitive against those, or the comparison to other hard-mining-based loss functions. The experiment results clarified some concerns, but need more efforts to be blended to the main paper.



Decision: Accept (Poster)