Title: Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training
Authors: Minguk Kang
Woohyeon Joseph Shim
Minsu Cho
Jaesik Park
Venue: NeurIPS 2021
Keywords: Generative Adversarial Networks
Conditional Image Generation
Adversarial Learning
Image Synthesis

Abstract: Conditional Generative Adversarial Networks (cGAN) generate realistic images by incorporating class information into GAN. While one of the most popular cGANs is an auxiliary classifier GAN with softmax cross-entropy loss (ACGAN), it is widely known that training ACGAN is challenging as the number of classes in the dataset increases. ACGAN also tends to generate easily classifiable samples with a lack of diversity. In this paper, we introduce two cures for ACGAN. First, we identify that gradient exploding in the classifier can cause an undesirable collapse in early training, and projecting input vectors onto a unit hypersphere can resolve the problem. Second, we propose the Data-to-Data Cross-Entropy loss (D2D-CE) to exploit relational information in the class-labeled dataset. On this foundation, we propose the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN). The experimental results show that ReACGAN achieves state-of-the-art generation results on CIFAR10, Tiny-ImageNet, CUB200, and ImageNet datasets. We also verify that ReACGAN benefits from differentiable augmentations and that D2D-CE harmonizes with StyleGAN2 architecture. Model weights and a software package that provides implementations of representative cGANs and all experiments in our paper are available at https://github.com/POSTECH-CVLab/PyTorch-StudioGAN.

TL;DR: We propose a new conditional GAN: ReACGAN that can enjoy stable training and exploit data-to-data relationships during training.

Review discussion:
Originality.
+ (+) Showing that the feature norm and the gradient (dL/dw) norm has connection to the performance is interesting.
+ (+)Combining the standard cross-entropy for classification and contrastive loss is novel (eq. 5).
+ (+)Difference between ReACGAN and ContraGAN is clearly described.
- (-) Normalizing the representations for computing inner product has already been standard in representation learning (simclr, mocov2, clip ...). They should be covered as related work.
Quality.
+ (+) Hyperparameter search is provided in the appendix. FID is mostly robust to the choices.
- (-) ACGAN's early training collapse is not rigorously defined and not shown either empirically or theoretically.
- (-) Incorrect argument: As $F(x_i)^Tw_k$ arises, p gets closer to 1, (1-p) gets closer to 0, and the norm of the gradient shrinks.
> L108 The norm of the gradient begins to explode as the inner-product value of $F(x_i)^Tw_k$ arises
- (-) The paper does not cover multi-task discriminator which has is a strong standard in recent methods [FUNIT, StarGANv2, Which training methods do actually converge]. Describing that the multi-task discriminator does not have similar phenomena will make the argument on ACGAN's instability stronger.
- (-) Reasons for using DiffAug and not using [ADA] are not described.
Clarity.
+ (+) The proposed method is clearly described enough to be implemented.
- (-) Thresholds m_p and m_n should be termed margins instead of thresholds, following [Learning shallow convolutional feature descriptors with triplet losses].
- (-) The proposed method is referred to as ReACGAN and Big+D2D-CE interchangeably. Using a single term is recommended to avoid confusion.
Significance.
+ (+) The proposed method is effective on ACGAN and ProjectionGAN variants.
+ (+) The proposed method synergies with consistency regularization.
Main reasons for my rating toward rejection come from logical weaknesses. Please refer to the <Quality> section.
No
Thank you for your insightful comments on our paper. Especially, we thank the reviewer’s comment on the lack of explanation about ACGAN’s early training collapse and how it occurs during training. This rebuttal responds to the reviewer’s concerns and comments below.
**C1. ACGAN's early training collapse is not rigorously defined and not shown either empirically or theoretically.**
The early training collapse refers to the phenomenon that the learning process becomes biased by a dominant objective at the early stage of training. ACGAN tends to undergo the early training collapse, where the classification objective dominates over adversarial learning and even stops the updates of the generator after a few early iterations.
For example, the FID value of ACGAN does not increase after 10K iterations on Tiny Imagenet while the other models (BigGAN [4], ContraGAN [22], and ours) have their best FID around 40K iterations. This has also been witnessed by other work [18, 19, 20, 24, 22] and visually or numerically shown in the related work of [22, 39, Ref 2]. We will clarify this by adding more explanations in our final manuscript.
Throughout the paper, we show why the undesirable training collapse occurs during ACGAN training in the analytic way (section 3.1).
Also, we experimentally identified that the large norm of features and poor classification ability of an early stage classifier causes an undesirable gradient exploding problem, which eventually leads to early training collapse (Figure 2). To remove this issue, we propose a normalization technique and demonstrate that the normalization can prevent ACGAN from collapsing at the early stage (please refer to Table 4 in our paper).
**C2. Incorrect argument: As F(x_{i})^{T}w_{k} arises, p gets closer to 1, (1-p) gets closer to 0, and the norm of the gradient shrinks.**
line 108--109:
As a result, the norm of the gradient begins to explode as the inner-product value arises, and the vector F (xi) stretches out to w_{k} direction.
Thanks for the good point. We find that our argument reads unclear due to missing details.
Although F(x_{i})^Tw_{k} arises at the early stage of training, it does not mean that p gets closer to 1 since inner products between F(x_{i}) and w_{j} is not close to 0 and not negligible. To clarify the argument, we here explain the undesirable behavior of ACGAN by reporting probabilities, norms of gradients, and feature vectors from an experimental result on the Tiny Imagenet dataset. In the experiment on Tiny Imagenet, the training collapse occurred around 4K iterations (Best FID = 106.410). At the early stage of training, it is very hard for the classifier to correctly predict the true labels of given images. In this circumstance, the feature vector F(x_{i}) tries to stretch out to w_{k} direction as training progresses, and the norm of F(x_{i}) continues to increase.
However, the problem is that probability p_{i,k} does not reach 1.0 since there exists w_j (j \in {1,...c} \ {k}) whose inner product value with F(x_{i}) is not negligible. This may cause unexpected large gradients for the classifier due to the low probability and large norm of the feature map.
As a result, the exploding gradient spoils adversarial training because ACGAN uses a shared network for adversarial and classification learning (please refer to Table R3). We will clarify this with  the experimental results in our final paper.
**Table R3**. Average norms of feature vectors (F(x)) and gradients in the ACGAN classifier [18] and the average value of target probabilities.
| Backbone: ACGAN [18]       | 1K iteration |    2K   |    3K   |    4K   | 5K      |
|----------------------------|:------------:|:-------:|:-------:|:-------:|---------|
| FID [32]                   |    153.421   | 126.691 | 112.142 | 106.410 | 107.973 |
| Average norm of features   |    23.759    |  28.951 |  42.963 |  62.191 | 95.851  |
| Average value of target probabilities |     0.013    |  0.029  |  0.086  |  0.130  | 0.245   |
| Average norm of gradients  |     0.055    |  0.072  |  0.097  |  0.123  | 0.241   |
**C3. The paper does not cover multi-task discriminator which has a strong standard in recent methods [FUNIT, StarGANv2, which training methods do actually converge].  Describing that the multi-task discriminator does not have similar phenomena will make the argument on ACGAN's instability stronger.**
Thanks for the constructive comment and suggestion.
First of all, we will cover multi-task discriminator with references of FUNIT [Ref 3] and StarGANv2 [Ref 4] and describe the relations to our algorithm.
Next, as the reviewer requested, we test the multi-task discriminator based on ACGAN architecture with a slight modification of losses and the last layers of the discriminator. We use Tiny Imagenet as our test benchmark and train the model with the same configuration and hyperparameters with ACGAN except for batch size; we use 256 instead of 1024 since it was more beneficial to the multiple discriminator.
Consequently, as noted by the reviewer, the training does not collapse at the early stage like ACGAN and also the final results are substantially improved from ACGAN (MD of Table R4) in accordance with the results of FUNIT and StarGANv2.
Furthermore, ACGAN with normalization performs much better than the multiple discriminator. All these results strongly support our claim that ACGAN has its unique problem, and the normalization is highly effective to address it as theoretically (section 3.1) and empirically (Figure 2 and C.2) proved in the paper. We will include these experiments in the final manuscript.
**Table R4**. Comparison with ACGAN [18] and multi-task discriminator [Ref 3] on Tiny Imagenet. We refer to the multiple discriminator as MD.
|          | ACGAN [18] | MD [Ref 3] | ACGAN + Normalization (proposed) | 
|----------|:----------:|:--------------------------------:|:----------:|
| IS [31]  |    6.00    |               9.79              |    **13.46**    |
| FID [32] |    96.04   |               43.38              |    **30.33**   |
**C.4 Reasons for using DiffAug and not using [ADA] are not described.**
ADA and DiffAug are popular augmentation methods for GAN training. Since training GAN with ADA requires additional hyperparameter tuning (target probability for augmentation and augmentation probability increase rate), we utilized DiffAug for our experiments to save hyperparameter tuning time for experiments. We will add this description in the revision.
**C.5 Normalizing the representations for computing inner product has already been standard in representation learning (simclr, mocov2, clip ...). They should be covered as related work.**
Thank you for bringing our attention to these studies! Definitely, our work has a close relation to these studies in a way that they all found \ell2-normalization beneficial to embed samples for its designated tasks.
In fact, \ell2-normalization has been introduced in a variety of tasks; face recognition [Ref 5], metric learning [Ref 6], contrastive learning [Ref 7-9], and multi-modal learning [Ref 10]. Our work also uses \ell2-normalization, but with a distinctive purpose to resolve the gradient exploding problem of ACGAN and improve the quality of generated samples.
Notably, our paper is the first work that explores the reason for the early training collapse of ACGAN and provides \ell2-normalization as a solution with theoretical (section 3.1) and experimental (Figure 2 and C.2) supports. We will reflect them in the final manuscript accordingly.
**C.6 Thresholds m_p and m_n should be termed margins instead of thresholds.**
Thanks for the suggestion. Following your advice, we will replace the "threshold" term by the “margin" one.
[Ref 2] Hou, Liang, et al. "cGANs with Auxiliary Discriminative Classifier." arXiv preprint arXiv:2107.10060 (2021).
[Ref 3] Liu, Ming-Yu et al. “Few-Shot Unsupervised Image-to-Image Translation.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019): 10550-10559.
[Ref 4] Karras, Tero et al. “Analyzing and Improving the Image Quality of StyleGAN.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 8107-8116.
[Ref 5] Wang, H. et al. “CosFace: Large Margin Cosine Loss for Deep Face Recognition.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018): 5265-5274.
[Ref 6] Kim, Sungyeon et al. “Proxy Anchor Loss for Deep Metric Learning.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 3235-3244.
[Ref 7] Chen, Ting et al. “A Simple Framework for Contrastive Learning of Visual Representations.” ArXiv abs/2002.05709 (2020).
[Ref 8] He, Kaiming et al. “Momentum Contrast for Unsupervised Visual Representation Learning.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 9726-9735.
[Ref 9] Khosla, Prannay et al. “Supervised Contrastive Learning.” ArXiv abs/2004.11362 (2020).
[Ref 10] Radford, Alec et al. “Learning Transferable Visual Models From Natural Language Supervision.” ICML (2021).
Thanks for your effort in the response.
**C1. ACGAN's early training collapse is not rigorously defined and not shown either empirically or theoretically.**
> The early training collapse refers to the phenomenon that the learning process becomes biased by a dominant objective at the early stage of training.
How do we judge whether the classification objective dominates the adversarial objective? Non-increasing FID is not enough for the judgement.
How do we judge the gradient has exploded? The gradients in Figure 2b does not diverge.
We have to be careful to choose expressions to make the arguments rigorous.
**C2. Incorrect argument**
It is not about being close to 0 or being negligible, but about getting smaller.
**C3. The paper does not cover multi-task discriminator**
Thank you for running additional experiments.
Is R1-regularizer [which training methods do actually converge] employed? It has the same goal, i.e., reducing the gradient norm. In their paper, they achieve IS 16 on ImageNet.
**C.4 Reasons for using DiffAug and not using [ADA] are not described.**
ADA provides an automatic way to compute proper probability.
**C.5 Normalizing the representations for computing inner product has already been standard in representation learning **
This response addresses my concern.
**C.6 Thresholds m_p and m_n should be termed margins instead of thresholds.**
This response addresses my concern.
Thanks for your additional comments on our work. We try to answer the remaining concerns C1, C2, and C3 below.
**C1-1. How do we judge whether the classification objective dominates the adversarial objective? Non-increasing FID is not enough for the judgment.**
We can identify whether ACGAN suffers from the early training collapse or not through two symptoms.
First, FID value of ACGAN does not decrease after a few early iterations. This issue implies that the adversarial training does not progress well, and the quality of generated images does not get better as the training continues.
Second, contrary to the trend of FID value, the classification error continues to decrease, which means the discriminator network becomes biased by the classification objective instead of the adversarial loss. We will add more analysis about this in our final manuscript.
**C1-2. How do we judge the gradient has exploded? The gradients in Figure 2b do not diverge.**
Thanks for pointing out this. The ACGAN's gradient curve in Figure 2 (b) shows that the gradient of the classifier increases very quickly (soaring) compared to its original scale at the early training stage, but the gradient does not diverge. We will refine this expression to make our argument more rigorous, but we would like to state that our main claim still holds. In our main paper, we analytically and experimentally showed that the unbounded feature maps can cause the soaring gradient problem at the early training stage.
Then, we empirically verify projecting feature maps onto a unit hypersphere can resolve this problem throughout our paper.
**C2. Incorrect argument (lines 108-109)**
Before responding to the reviewer's concern, we bring our arguments in lines 106-110.
``` lines 106-110
In the early training stage, the classifier is prone to make incorrect predictions, resulting in low probability outputs. This phenomenon gets severe as the number of categories of the dataset increases.
As a result, the norm of the gradient begins to explode as the inner-product value of F(x_{i} )^T w_{k} arises, and the vector F (x_i) stretches out to w_{k} direction. This may break the balance between adversarial and classification training and can lead to early training collapse.
In our main paper, we stated that the classifier at the early training stage is prone to make incorrect predictions.
Although the classifier’s gradient can be shrinked as the probability p increases, the norm of the feature vector F(x_i) increases much faster, so the classifier’s gradient soars at the early training stage.
Therefore, we would like to state that the argument "the norm of the gradient begins to explode as the inner-product value of F(x_{i} )^T w_{k} arises, and the vector F (x_i) stretches out to w_{k} direction" is correct since the low probability value (p_{i,k}) can not shrink the norms of feature maps. We will clarify this in our final manuscript. Thank you.
**C3. Is R1-regularizer [which training methods do actually converge] employed? It has the same goal, i.e., reducing the gradient norm. In their paper, they achieve IS 16 on ImageNet.**
Yes, the result is obtained with the R1-regularizer, and it gives a slight improvement over the not-applied one in our experiments. We think that the discrepancy between our results and the reported score comes from two reasons: 1) the measurement procedures are different; StarGANv2 [Ref 11] did not use IS for the evaluation metric, and FUNIT [Ref 3] measured IS using the newly-trained Inception networks with their datasets, 2) the evaluation datasets are different; we evaluate the method on Tiny_Imagenet [30], but they use another variant that extracts only the face parts using a detector from the animal classes.
However, the results (Table R4) show ACGAN performs worse than the multi-task discriminator. This observation implies that the multi-task discriminator might not suffer from the same issue of ACGAN. More importantly, the normalization in ACGAN appears much more effective than using the multi-task discriminator, which validates our claim that the issue of ACGAN can largely be alleviated using our method.
[Ref 11] Choi, Yunjey, et al. "Stargan v2: Diverse image synthesis for multiple domains." 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR): 8188-8197.
### Clarity
This paper claims to Reboot ACGANs but in the paper the best performance is achieved by D2D-CE loss which is very similar to the contrastive loss used in contraGAN. The auxiliary classifier in D2D-CE performs inferior to the ACGAN classifier as shown in Appendix Table A9, which clearly shows that the discriminator with D2D-CE loss is definitely not an auxiliary classifier (as it has poor classification performance compared to ACGAN). I do appreciate the finding that normalization of embedding helps in improvement of ACGAN, but the final model is clearly dissimilar to ACGAN.
Hence in my opinion the claim of rebooting ACGAN is a bit misleading.
### Originality:
I find that a lot of ideas in this paper have already been explored in the ContraGAN [K1] paper
1. Idea of Normalization: The idea of normalization of embeddings such that they lie on a sphere is also used by ContraGAN.
2. Idea of Positive Negative Repulsion: The idea of introducing a positive negative based repulsion seems similar to the contrastive loss in ContraGAN.
3. Eq.5 in the paper is very similar to Eq. 7 in ContraGAN paper which is the base equation both papers build on.
4. The authors have introduced D2D-CE which seems to be inspired from hard mining based contrastive losses and afaik is novel in GANs.
The works in appendix references [25,26,27] already have introduced the hard mining losses for classification, hence I recommend to use these losses as well and show a comparison, to make the submission more stronger.
### Quality
There have been a lot of experiments done for verifying the effectiveness of the proposed method. I find that gains in performance for the CIFAR-10 and Tiny Imagenet overlap in the standard deviation intervals, hence the claim of state-of-the-art cannot be fully verified on these datasets. The technique seems to perform better in case of the CUB dataset.
### Significance
I find that a lot of the ideas stated in this paper have already been explored in ContraGAN paper.
Also the proposed loss is in my opinion, is more similar to contrastive loss rather than ACGAN loss. The description of the current work as Rebooting ACGAN seems misleading as the D2D-CE method is a contrastive method based on hard-mining of samples.
[K1] Kang M, Park J. Contragan: Contrastive learning for conditional image generation. arXiv preprint arXiv:2006.12681. 2020 Jun 23.
No
We appreciate the thorough review regarding our study. By reading the review carefully, we have found that the reviewer’s concerns are mainly from two sources: (1) some ideas in our paper have already been introduced in ContraGAN paper [22], and (2) D2D-CE seems to be more like a contrastive loss than the cross-entropy loss used in ACGAN. In this rebuttal, we address these concerns below:
**C1. Some ideas in our paper have already been explored in the ContraGAN [22] paper.**
As the reviewer pointed out, our work has some apparent similarities with ContraGAN in two aspects: feature normalization and embedding loss. The details, however, turn out to be different as follows.
**[Feature normalization]**
While ContraGAN [22] utilizes the feature normalization without any explanation or verification of its necessity, our work, ReACGAN investigates the problem of early training collapse of ACGAN and then introduces a simple yet effective normalization technique as its solution, which is supported by theoretical analyses (section 3.1) and experimental results (Figure 2 and C.2). We believe this in-depth verification of the normalization technique is a novel technical contribution of ReACGAN, which is distinct from ContraGAN and others.
**[Pushing and pulling feature embeddings]**
While the formulation of D2D-CE loss has a similar form with that of ContraGAN’s 2C loss, they are very different in the perspective of sample mining. The 2C loss has multiple similarities between an anchor sample and its positive samples in both the numerator and denominator terms. According to the authors’ explanation, this formulation is introduced to eliminate an unnecessary repulsion force caused by false-negative samples reported by [A25, A26, A27].
However, as can be seen in appendix E, the similarities between an anchor and its positive samples in the numerator of 2C loss can cause undesirable easy positive mining.
Moreover, the false negative terms in the denominator of 2C loss can attenuate the negative repulsion force. These characteristics of 2C loss can cause the embedding features of the discriminator to be clustered in an unintended way and induce a class confusion problem.
In contrast, our D2D-CE loss is relatively free from the class confusion problem. This is because D2D-CE loss does not have similarities between an anchor and positive samples both in the numerator and denominator. We mathematically and experimentally verify the effectiveness of D2D-CE compared to 2C loss in the mining perspective.
As a result, We argue that D2D-CE is a novel objective that can alleviate training instability caused by softmax cross-entropy in ACGAN and can effectively consider relational information between data points.
As a result, Top-1 classification accuracy of generated images is only 2.866% for ContraGAN, while ReACGAN's accuracy is over 20%.
Please check appendix E (line 101--104) and appendix H in our supplement to see the discussion of the difference between D2D-CE loss and 2C loss from a mathematical perspective.
**C2. The description of the current work as Rebooting ACGAN seems misleading as the D2D-CE method is a contrastive method based on hard-mining samples.**
As noted in lines 94-121, our method, ReACGAN, has been motivated and developed to resolve the early collapsing issue of ACGAN. One may also think ReACGAN is an effective variant of ContraGAN in the perspective of sample mining.
We believe that the current name, ReACGAN, is not misleading for the following reasons.
First, the origin of D2D-CE is a softmax cross-entropy loss used in ACGAN. We start by analyzing the analytic gradient value in ACGAN’s classifier and adopt considering data-to-data relationships to further boost the generation performance of ACGAN.
Second, ReACGAN also has an auxiliary classifier (linear layer) for discriminating class labels of data. For these reasons, we think the description of the current work as Rebooting ACGAN logically makes sense.
**C3. The auxiliary classifier in D2D-CE performs inferior to the ACGAN classifier, as shown in Appendix Table A9, which clearly shows that the discriminator with D2D-CE loss is definitely not an auxiliary classifier.**
We believe there is a misunderstanding here. As can be seen in Table 2 (high IS, but relatively low FID values of ACGAN) and Figure H.6 in the appendix, ACGAN tends to generate easily classifiable samples without considering diversity and fidelity. The relatively better classification accuracy of ACGAN compared to ReACGAN (shown in Table A9) is a consequence of generating easily classifiable samples rather than the sign of a better classifier.
Therefore, the relatively low classification accuracy of ReACGAN compared to ACGAN does not indicate that ReACGAN is not a variant of an auxiliary classifier GAN.
**C4. The works in appendix references [A25,A26,A27] already have introduced the hard mining losses for classification, hence I recommend to use these losses as well and show a comparison, to make the submission stronger.**
Thank you for your valuable suggestion. The work [A25, A26, A27] deals with removing the undesirable repulsion force caused by false negative samples without any label supervision. Since [A25, A26, A27] utilizes a self-supervised version of contrastive loss (InfoNCE) [Ref 1], and ReACGAN utilizes label information for conditioning, it is not possible to adopt the InfoNCE loss to ReACGAN directly.
Instead, we borrow the idea of removing the effect of false negatives proposed by [A25] in this rebuttal.
We abbreviate the false-negative mask used in ReACGAN as FNM and the method for relieving the negative repulsion force [A25] as Debias. As can be seen in Table R2, ReACGAN gives slightly better results than ReACGAN - FNM + Debias on CIFAR10 and Tiny_imagenet datasets. This is because ReACGAN directly utilizes class label information to remove false negatives in the denominator, while ReACGAN - FNM + Debias does this in an unsupervised way.
**Table R2**. FID values (lower is better) of ReACGAN and its modifications for dealing with false-negative samples. FNM indicates false negatives mask proposed by us (please refer to Figure 3). Debias means the idea for relieving the negative repulsion force used in [A25].
|               | ReACGAN - FNM | ReACGAN - FNM + Debias [A25] | ReACGAN |
|:-------------:|:-------------:|:----------------------------:|:-------:|
|    CIFAR10    |     8.143     |              7.99            |   **7.79**  |
| Tiny_Imagenet |       -       |             27.93            |   **26.94** |
|     CUB200    |       -       |             114.59           |   **15.18** |
**C5. gains in performance for the CIFAR-10 and Tiny Imagenet overlap in the standard deviation intervals**
We conducted most experiments three times and underlined the numbers within one standard deviation compared to the best model. Based on FID value, the most widely adopted metric for evaluating GANs, ReACGAN outperforms other baselines without overlapping the one deviation interval in most cases (refer to Table 1, 2, appendix D, and the authors’ response for reviewer ojXE more details).
**C6. More discussion on additional training time due to the new loss.
Also some things like how much time was required for choosing hyperparameters can also be discussed in limitations.**
We investigate the time spent to train each model on Imagenet using eight Nvidia V100 GPUs. The batch size is set to 2048. We find that ReACGAN brings in a slight computational overhead and takes about 1.05~1.1 longer time than other algorithms.
Specifically, BigGAN takes 17m 37s, ContraGAN 18m 24s, ReACGAN 18m 52s per 100 generator updates. For hyper-parameter selection, we can recommend using one of our pre-searched parameters like \tau=0.5 and \m_n=0.95 since our method offers robustness across datasets, as shown in Appendix B.
[Ref 1] Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).
I have read the rebuttal, below are my major concerns which are still unanswered
Normalization: The normalization has shown to have better performance (already) for the classification tasks (as shown in SimCLR paper (https://arxiv.org/pdf/2002.05709.pdf)). The Contra GAN paper also used that same loss which was effective.
Hence, I am not convinced about the novelty of the analysis.
ReACGAN: In this case I still don’t understand why the increase from 2.866% to more than 20% is advantageous for D2D-CE.
But if we consider ACGAN loss we get accuracy of around 70% but the performance of GAN doesn’t improve (as the authors have mentioned).
Hence, I think the paper still requires some more analysis on how well the classifier should perform for getting the best GAN.
Also, the classifier in D2D-CE loss has an accuracy of 20% which is very less in comparison to ACGAN (hence I am still not convinced that this work should be called RebootingACGAN)
We appreciate the reviewer using his/her precious time to give additional feedback for improving our work. In this response, we politely request the reviewer to consider the novelty of our work and the origin of ReACGAN one more time.
**[Novelty of feature normalization]**
As the reviewer stated, the normalization has been used in the self-supervised learning community (MoCo [Ref 8], SimCLR [Ref 7], etc.) and ContraGAN paper.
However, we want to emphasize that the motivation and effects of the normalization technique used in ReACGAN and those studies are different. MoCo and SimCLR use the normalization trick to improve a neural network's representation quality without label supervision. ContraGAN utilizes the normalization technique to adopt a metric learning loss that can consider data-to-data relationships.
However, the normalization trick used in ReACGAN is devised to alleviate the early training collapse problem, and we demonstrated normalizing features can prevent ACGAN from collapsing at the early stage of training.
In short, the form of normalization is the same, but the motivation and its purpose are distinct.
**[Large improvement over ContraGAN]**
The normalization technique can alleviate the early collapsing issue and significantly improve the generation quality of ACGAN, but ReACGAN is not limited to these. As described in Sec 3.4 with the elaboration in Appendix E and F, ReACGAN incorporates data-to-data relationships with more appealing properties than ContraGAN, where ContraGAN suffers from easy positive mining and attenuated repulsion force to the negative.
Therefore, we conjecture that the increase in performance reveals these distinctions, making ReACGAN superior to ContraGAN, even though the same normalization technique is applied.
**[Accuracy of ACGAN and ReACGAN]**
**1.
Hence, I think the paper still requires some more analysis on how well the classifier should perform for getting the best GAN.**
Thanks for your suggestion. We will conduct more experiments to analyze the relationship between classification accuracy and generation performance in the future.
**2. The classifier in D2D-CE loss has an accuracy of 20%, which is very less in comparison to ACGAN (62.412%) (hence I am still not convinced that this work should be called RebootingACGAN).**
As the reviewer commented, ACGAN results in high classification accuracy (62.412%), Inception Score (62.99) but shows low FID (26.35). These scores indicate ACGAN generates easily classifiable images, as shown in Figure H. 11, Figure H. 15, and Figure H. 19 in our appendix.
Therefore, the outstanding classification accuracy of ACGAN results from the classifier's biased training toward generating easily classifiable images instead of estimating the data distribution p(x,y). In this circumstance, we insist that the relatively low classification accuracy of ReACGAN compared to ACGAN does not indicate ReACGAN is not a descendent of ACGAN. The high accuracy of ACGAN originates from the undesirable behavior of ACGAN.
Thus, an improved ACGAN does not necessarily yield better or comparable classification accuracy than ACGAN.
Although generating photo-realistic, diverse, and well-conditioned images is ideal and the ultimate aim of conditional image generation, it is interesting to know that existing cGANs do not meet all conditions: realism, diversity, and image conditioning. We will add this discussion in the paper.
In addition to this, we want to state that ReACGAN can be regarded as a variant of ACGAN in the perspective of its origin. ReACGAN is built upon an auxiliary classifier and the cross-entropy loss.
Moreover, the ReACGAN follows the same training procedure used in ACGAN.
Paper motivates with identifying the gradient explosion problem especially in the early training phase in the ACGAN due to the auxiliary loss and convincingly presents the idea to tackle the problem.
The paper is well written and easy to follow. All the notations, facts, and figures are clear to understand.
The experiments are extensive and the theoretical justifications seem correct.
No
We want to thank the reviewer for positive comments on our work. We feel grateful that the reviewer found our paper clear, justifiable, and superior to existing baselines. As suggested by other reviewers (HH86, t8Ys), we will provide additional experiments regarding debiasing loss for false negative removal (Table R2 in authors' response to reviewer HH86) and multi-task discriminator (Table R4 in authors' response to reviewer t8Ys).
In this paper, the authors analyse why ACGAN training becomes unstable as the number of classes increases.
Then they propose how to solve this problem :
1) by analytically calculating gradients of the softmax cross-entropy loss used in ACGAN. By examining the exact values of analytic gradients, they show that the unboundedness of input feature vectors and poor classification performance in the early training phase can cause an undesirable gradient exploding problem;
2) having alleviated the instability, they propose the Rebooted Auxiliary Classifier Generative Adversarial Networks (ReACGAN) using the Data-to-data Cross-Entropy loss (D2D-CE). ReACGAN projects image embeddings and proxies onto a unit hypersphere and computes similarities for data-to-data and data-to-class consideration.
Additionally, they introduce two threshold values for inter-class and intra-class separability. In this way, ReACGAN overcomes the training instability and can exploit supervisory signals by explicitly considering data-to-class and data-to-data relationships, and also by implicitly looking at class-to-class relationships in the same mini-batch.
The authors do indeed identify and resolve a major problem of ACGAN but their contribution to the problem is incremental in the sense that when compared with the existing state of the art methods, their performance is similar.
No
We appreciate the reviewer for taking time to understand the gist of our paper and providing valuable feedback. Below, the response to each concern is presented one by one.
**C1. “When compared with the existing state of the art methods, their performance is similar.”**
**Table R1**. Relative FID [32] improvements (%) of ReACGAN compared to SOTA GANs. *: Although SAGAN [26] is not considered a cutting-edge model, we could not find a study that reports a better FID value than SAGAN [26] for ImageNet experiment using a batch size of 256.
|                                     |      CIFAR10 [27]     |     Tiny-Imagenet [30]     |      CUB200 [29]      |      Imagenet [28]      |
|:-----------------------------------:|:---------------------:|:--------------------------:|:---------------------:|:-----------------------:|
| Previous best (without Diffaug [9]) | BigGAN [4] (FID=8.08) | ContraGAN [22] (FID=28.55) |   BigGAN (FID=18.30)  | SAGAN [26]* (FID=18.28) |
|               ReACGAN               |         +2.48%        |           +5.08%           |        +15.85%        |         +20.19%         |
|   Previous best  (with Diffaug [9])  |   BigGAN (FID=7.17)   |    ContraGAN (FID=15.40)   | ContraGAN (FID=11.02) |    BigGAN (FID=18.57)   |
|        ReACGAN + Diffaug [9]        |         +5.30%        |           +7.47%           |        +15.88%        |         +34.03%         |
While the improvements might appear small in some metrics, the improvements in FID, which is the most widely adapted to evaluate generative models, range from 2.48% to 34.03% compared with previous SOTA models (SAGAN [26], BigGAN [4], and ContraGAN [22]). This is a significant improvement when it comes to complex and high-resolution datasets (i.e., CUB200 [29] and Imagenet [28]).
Moreover, we want to emphasize that the improvements in Imagenet are achieved with fewer iterations than other competitive models (please refer to Figure 2 in Tab. 2 in the main paper).
**C2. “It would be useful to see the performance of algorithms for different values, as well -- ie to see how robust it is wrt different values.”**
Please note that we have investigated the robustness of ReACGAN in appendix B and found that ReACGAN is indeed tolerant to hyperparameter selections for different temperatures, positive and negative threshold values.
**C3. “In Table 1 and 2, what does the underlying mean?”**
The underlines are drawn for the numbers within 1-standard deviation from the best performed one, where the standard deviation is calculated by three independent experiments using three different random seeds. We will make it clear in the revised manuscript.
**C4. “None of the algorithms attains IS close to that of the real data at the Tiny-Imagenet dataset; do the authors have any explanation about that?”**
Training dataset of Tiny-Imagenet contains 100K 64x64 RGB images and is categorized into 200 classes. Since Tiny Imagenet has only 500 images per class, it is very hard for GANs to achieve a high IS without using augmentations for data-efficient training.
**C5. “Given that the paper contains graphs, I would advise the authors to employ colourblind-friendly colours so that they can be accessible from colour-blind fellows.”**
Thanks for the advice. We will change the colored figures with textured figures so that they become colorblind-friendly.
Dear Authors,
After reading the paper, the reviews of all the reviewers, your rebuttals and the corresponding discussions I will stick with my original opinion "5-Marginally below the acceptance threshold" as there are still some points yet to be addressed and then re-discussed as other reviewers pointed out, as well.
Yours,
Reviewer ojxe

Meta-review:
role: PC
data: comment: The authors identify that the auxiliary classifier GAN (ACGAN) fails to scale to more classes and is biased towards easily classifiable examples because ACGAN can suffer from gradient explosion when training the classifier, causing the collapse of training. They then propose to normalize the feature vectors and add a data-to-data loss function to improve the performance. The resulting model is named Rebooted ACGAN (ReACGAN), which can be easily combined with other techniques like consistency regularization for better performance. Experimental results reveal some competitive performance of ReACGAN.

The reviewers generally appreciate the clarity of the paper and the convincing argument from the claimed issue of ACGAN to the improved model. One issue that could be addressed easily is to rigorously define and describe what the authors mean by early training collapse. The major issue that was not easily addressed is on how the authors can establish originality with respect to

(1) the widely-used normalization technique in other GANs and representative learning works. The authors argue that their in-depth investigation, instead of the more ad-hoc use of other works, is an important contribution. While the contribution is indeed of some value, the authors only position this investigation within the narrow angle of the ACGAN model. It is not clear whether the contribution can also be generally used to explain the normalization technique used in other GANs. This narrow angle limits the impact of the contribution.

(2) the ContraGAN work which has a contrastive loss that shares some similarity to the proposed data-to-data loss function. While some reviewers believe that the difference between ReACGAN and ContraGAN is clearly described, and the authors argue that ReACGAN and ContraGAN losses are designed with different principles (Line 177), other reviewers point out that more efforts are needed to highlight the differences between the two loss functions, such as empirical studies on how "false negative samples" or other loss variants affect performance (like the preliminary results provided in the rebuttal). While the authors' arguments on how ReACGAN originated from ACGAN (rather than ContraGAN) is convincing, they do need more efforts to clarify whether the ReACGAN is significantly different to ContraGAN in their final form to claim for originality. Some efforts are also needed to smoothly incorporate the rebuttal paragraphs into the revised paper.

One issue that some reviewers and authors disagree on is whether ReACGAN should be "called" ReACGAN, given that the classifier nature in ACGAN was deliberately weakened for ReACGAN and the proposed loss makes the whole model look a lot more like ContraGAN. The naming issue is arguably more subjective and is not taken as the key factor behind the final decision making. Somehwo the authors are encourage to keep studying a related issue on analyzing how the classifier performance affects conditional generation performance.

The authors addressed some of the concerns with preliminary experiments, such as the comparison to multi-task discriminators to ensure whether ReACGAN is competitive against those, or the comparison to other hard-mining-based loss functions. The experiment results clarified some concerns, but need more efforts to be blended to the main paper.




Decision: Accept (Poster)