Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training
Authors: Minguk Kang
Woohyeon Joseph Shim
Minsu Cho
Jaesik Park
Venue: NeurIPS 2021 Poster (2021)
Keywords: Generative Adversarial Networks
Conditional Image Generation
Adversarial Learning
Image Synthesis

Abstract: Conditional Generative Adversarial Networks (cGAN) generate realistic images by incorporating class information into GAN. While one of the most popular cGANs is an auxiliary classifier GAN with softmax cross-entropy loss (ACGAN), it is widely known that training ACGAN is challenging as the number of classes in the dataset increases. ACGAN also tends to generate easily classifiable samples with a lack of diversity. In this paper, we introduce two cures for ACGAN. First, we identify that gradient exploding in the classifier can cause an undesirable collapse in early training, and projecting input vectors onto a unit hypersphere can resolve the problem. Second, we propose the Data-to-Data Cross-Entropy loss (D2D-CE) to exploit relational information in the class-labeled dataset. On this foundation, we propose the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN). The experimental results show that ReACGAN achieves state-of-the-art generation results on CIFAR10, Tiny-ImageNet, CUB200, and ImageNet datasets. We also verify that ReACGAN benefits from differentiable augmentations and that D2D-CE harmonizes with StyleGAN2 architecture. Model weights and a software package that provides implementations of representative cGANs and all experiments in our paper are available at https://github.com/POSTECH-CVLab/PyTorch-StudioGAN.

Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training
Minguk Kang
Woohyeon Shim
Minsu Cho
Jaesik Park
Pohang University of Science and Technology (POSTECH), South Korea {mgkang, wh. shim, mscho, beesik.parke @postech. paros ac.kr
Abstract
Conditional Generative Adversarial Networks (cGAN) generate realistic images by incorporating class information into GAN. While one of the most popular cGANs is an auxiliary classifier GAN with softmax cross-entropy loss (ACGAN), it is widely known that training ACGAN is challenging as the number of classes in the dataset increases. ACGAN also tends to generate easily classifiable samples with lack of diversity. In this paper, we introduce two cures for ACGAN. First, we identify that gradient exploding in the classifier can cause an undesirable collapse in early train- ing, and projecting input vectors onto a unit hypersphere can resolve the problem. Second, we propose the Data-to-Data Cross-Entropy loss (D2D-CE) to exploit propose relational information in the class-labeled dataset. On this foundation, the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN). The experimental results show that ReACGAN achieves state-of-the-a -ftt----t---tttt generation results on CIFAR1O 10, Tiny- ImageNet, CUB200, and ImageNet datasets. We also verify that ReACGAN benefits from differentiable augmentations and that D2D-CE harmonizes with StyleGAN2 architecture. Model weights and a software package that provides implementations of representative eGANs and all experiments in paper are available at
Torch-StudioGAN,
Introduction
Generative Adversarial Networks (GAN) [1] are known the forefront approach to generating high-fidelity images of diverse categories [2, 3, 4, 6, 7, 8, 9]. Behind the sensational generation ability of GANs, there has been tremendous effort to develop adversarial objectives free from the vanishing gradient problem [10, 11, 12], regularizations for stabilizing adversarial training [11, 13, 14, 3. 15, 16, 17], and conditioning techniques to support the adversarial training using category information of the dataset [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28). Subsequently, the conditioning techniques have become the de facto standard for high-quality image generation. The models with the conditioning methods are called conditional Generative Adversarial Networks (cGAN), and cGANs can be divided into two groups depending on the discriminator's conditioning way: classifier-based GANs [18, 20, 23, 25, 27] and projection-based GANs [19, 3, 28].
The classifier-based GANs facilitate an auxiliary classifier to generate class-specific images by penalizing the generator if the synthesized images are not consistent with the conditioned labels. ACGAN [18] has been one the widely used classifier- based GANs for its simple design and satisfactory generation performance. While ACGAN and pulling classifier's s weights (proxies) against image embeddings [23]. it is well known that ACGAN training is prone to collapsing the early stage of training the number of classes increases [19, 23, 25, 27]. In addition, the generator of ACGAN tends to generate easily classifiable images the cost of reduced diversity [18, 19, 27]. Projection- baased GANs, the other hand, have shown cutting-edge generation results on datasets with a large number of categories. SNGAN [3],
exploit class information by pushing
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Rd
Ra
Positive
Whird *
Wiird
S4
ddg
dog
w dog
Positive
Easy negative
#lowerr
W flower
mp
My
(a) ACGAN [18]
(c) MHGAN [26]
R
P dog
g+
Reference
ddog
Negative
Hard negative
Projection-based
(d) ContraGAN [23)
(e) ReACGAN (Ours)
GAN [19]
Figure 1: Schematics that depict how cGANS perform conditioning. The red color means positive samples/proxy, and the blue color indicates negative samples/proxies. Arrows represent push-pul forces based on the reference sample. The length of an arrow indicates the magnitude of the force.
SAGAN [29], and BigGAN [4] are representatives in this family and can generate realistic images on CIFARIO 10 [30] and ImageNet [31] datasets. However, projection-based GANs only consider pairwise relationship between an image and its label proxy (data-to-class relationships). As the result, the projection-basee based GANs can miss an additional opportunity consider relation information between data instances (data-to-data relationships) as discovered by [23].
In this paper, analyze why ACGAN training becomes unstable the number of classes increases and propose remedies for (1) the instability and (2) the relatively poor generation performance of ACGAN compared with the projection- based models. First, begin by analytically deriving the gradient of the softmax cross- cross-entropy loss used in ACGAN. By examining the exact values of analytic gradients, discover that the unboundedness of input feature vectors and poor classification performance in the early training stage can cause an undesirable gradient exploding problem. Second, propose the Rebooted Auxiliary Classifier Generative Adversarial
with alleviating the instability, Networks (ReACGAN) using the Data-to-Data Cross-Entropy loss (D2D-CE). ReACGAN projects image embeddings and proxies onto unit hypersphere and computes similarities for data-to-data and data-to-class consideration. Additionally, we introduce two margin values for intra-class variations and inter-class separability. In this way, ReACGAN overcomes the training instability and can exploit additional supervisory signals by explicitly considering data-to-clate and data-to aata-to-data relationships, and also by implicitly looking at class-to-class relationships in the same
To validate our model, we conduct image generation experiments on CIFAR1O [30]. Tiny- ImageNet [32], CUB200 [33], and ImageNet [31] datasets. Through extensive experiments, demonstrate that ReACGAN beats both the classifier- based and projection-based GANs, improving the state of the art by 2.5%, 15.8%, 5.1%, and 14.5% in terms of Fréchet Inception Dis- tance (FID) [34] on the four datasets, respectively. We also verify that ReACGAN benefits from consistency regularization [16] and differentiable augmentations [9, 8] for limited data training. Finally, we confirm that D2D-CE harmonizes with the StyleGAN2 architecture [7].
Background: Generative Adversarial Networks
Generative Adversarial Network (GAN) [1] is an implicit generative model that aims to generate a
sample indistinguishable from the real. GAN consists of two networks: Generator G Z X
that tries to map a latent variable z ~ p(z) into the real data space X' and a Diseriminator D X -
[0, that strives to discriminate whether given sample is from the real data distribution Preat (x) from the implicit distribution Pgen (G(z) derived from the generator G(z). The objective a vanilla
GAN [1] can be expressed follows:
min max Ex-mp(x) (lw.lleooooooooo [Dx))) + En- (og (D G D
(1)

---

While GANs have shown impressive results in the image generation task [2. 35, 11, 13], training GANs often ends up encountering mode-collapse problem [36, 11 12]. As one of the preseriptions for stabilizing and reinforcing GANs, training GANs with categorical information, named conditional Generative Adversarial Networks (cGAN), is suggested [37, 18, 19]. Depending on the presence of explicit classification losses, cGAN can be divided into two groups: Classifier- GANs [18, 20, 23, 25, 27] and Projection-base GANs [19, 3, 4, 28]. One of the widely used classifier-based GANs is ACGAN [18], and ACGAN utilizes softmax cross-entropy loss to perform classification task with adversarial training. Although ACGAN has shown satisfactory generation results, training ACGAN becomes unstable as the number of classes in the training dataset increases [19, 23, 25, 27]. Besides, ACGAN tends to generate easily classifiable images at the cost of limited diversity [18. 19, 27]. To alleviate those problems, Zhou et al. [38] have proposed performing adversarial training on the classifier. Gong el al. [20] have introduced an additional classifier to eliminate a conditional entropy minimization process in the adversarial training. However, ACGAN training still suffers from the early-training collapse issue and the reduced diversity problem when trained on datasets with a large [32] and ImageNet [31].
number of class categories, such
In these circumstances, Miyato et al. [19] have proposed a projection discriminator for cGANs and have shown significant improvement in generating the ImageNet dataset. Motivated by the promising result of the projection discriminator, many projection-basee GANs [3. 29, 4, 16, 6, 17] have been proposed and become the standard for conditional image generation. In this paper, revisit ACGAN and unveil why ACGAN training is so unstable. Coping with the instability, we propose the Rebooted Auxiliary Classifier GANs (ReACGAN) for high-quality and diverse image generation.
3 Rebooting Auxiliary Classifier GANs
3.1 Feature Normalization
To uncover a nuisance that can cause the instability of ACGAN, we start by analytically deriving the gradients of weight vectors in the softmax classifier. Let the part the discriminator before the fully connected layer be a Feature extractor F X - F E ı2d and let Classifier C F - R be a single fully connected layer parameterized by W = [W].... wel E Raxe, where e denotes the number of classess. We sample training images X = {œ1, ..... NN } and integer labels = {yI..... } from the joint distribution p(x,y) Using the notations above, we can express the empirical cross-entropy loss used in ACGAN [18] as follows:
1 N LCE = - N
exp (F(x:) wy,
Σ log
Σラー exp (T(::))T)) Two,)
Based on Eq. (3), we can derive the derivative of the cross-entropy loss, w.r.t (1....... as follows:
OCcr 1 N
Σ 1=1
F(a.)
11 Pit))
Ow:
N
where 1y is an indicator function that will output 1 if ys = is satisfied, and pt.k is a class probability that represents the probability that i-th sample belongs to class k. mathematically (xp,)( mnx
>e= (x. w The equation above implies that the norm of the gradient of the softmax cross-entropy loss is coupled with the norms and directions of each input feature map F(x;)) and the class probabilities. In the early training stage, the classifier is prone to making incorrect predictions, resulting in low probabilities. This phenomenon occurs more frequently as the number of categories in the dataset increases. As the result, the norm of the gradient Pwry begins to explode as the vector F(x;) stretches out to we direction but being located close to the the other vectors (1) This often breaks the balance between adversarial learning and classifier training, leading to an early-training collapse. Once the early-training collapse occurs, ACGAN training concentrates on classifying categories of images instead of discriminating the authenticity of given samples. We experimentally demonstrate that the average norm of ACGAN's input feature maps increases as the training progresses (Fig. 2a). Accordingly, the average norm of the gradients increases sharply at the early training stage and decreases with the high class probabilities of the classifier (Fig. 2b, Fig. A3 Appendix). While the average norm of gradients decreases at some point, the FID value of ACGAN does not decrease, implying the collapse of ACGAN training (Fig. 2c).
@Ccc
3

---

Average nomes of impat frature maaee
Averuay norms of grodions of classilie
Thends of FID scvess
FID (loner in bester))
Norn of fouture
Norre of gralltee
A01-1
Schmei Namoalle
61
DMD Seggreccon
D2D-CE
0.30
16
0.301
Bletation
Iteration
Revniionn
(a) Feature norm
(b) Gradient norm
(c) FID score
Figure 2: Merits of integrating feature normalization and data-to-data relationship consideration. All experiments are conducted on Tiny-ImageNer [32] dataset. (a) Average norms of input feature maps F(-), (b) average norms of gradients of classification losses, and (c) trends of FID scores. Compared with ACGAN [18], the proposed ReACGAN does not experience the training collapse problem caused by excessively large norms of feature maps and gradients at the early training stage. In addition, ReACGAN can converge to better equilibrium by considering data-to-data relationships with easy positive and negative sample suppression.
As one of the remedies for the gradient exploding problem, we find that simply normalizing the F(a.)
feature embeddings onlo unil hypersphere [[[(,,]]] effectively resolves ACGAN's early-training collapse (see Fig. 2). The motivation is that normalizing the features onto the hypersphere makes the norms of feature maps equal to 1.0. Thus, the discriminator does not experience the gradient exploding problem. From the next section, we will deploy a linear projection layer P on the feature extractor F. And, we will normalize both the embeddings from the projection layer and the weight vectors (w),... ..., w.) in the classifier since normalizing both the embeddings and the weight vectors does not degrade image generation performance. We denote the normalized embedding P(F(2.)) [P(F(,))]]
as fü and the normalized weight vector ||,,
Wi as Byy-
3.2 Data-to-Data Cross-Entropy Loss (D2D-CE)
We expand the feature normalized softmax cross-entropy loss described in Sec. 3.1 the Data-to-Data Cross-Entropy (D2D-CE) loss. The motivations are summarized into two points: (1) replacing data- to-class similarities in the denominator of Eq. (3) with data-to-data similarities and (2) introducing two margin values to the modified softmax cross-entropy loss. We expect that point (1) will encourage the feature extractor consider data-to-class as well as data-to-data relationships, and that point (2) will guarantee inter-class separability and intra-class variations in the feature space while preventing ineffective gradient updates induced by easy negative and positive samples. To develop the feature normalized cross- cross-entrops entropy loss into D2D-CE, we replace the similarities between a sample embedding
and all proxies except for the positive one in the denominator, (). ....)) exp (ff, 0y) with similarities between negative samples in the same mini-batch. The modified cross-entropy loss can be expressed as follows:
N Σ 2=1
exp (f, Uy./T) exp (f EjeN(o exp (f. (/T)))
fCF N log
(5)
where 7 is a temperature, and (i)) is the set of indices that point locations of the negative samples whose labels are different from the reference label vy, the mini-batch. The self semilaimmlarity matrix of samples in the mini-batch used calculate the similarities between negative samples f. fjeN() with a false negative mask (see Fig. 3). Thus, Eq. (5) enables the discriminator to contrastively compare visual differences between multiple images and can supply more informative supervision for image conditioning. Finally, we introduce two margin hyperparameters to E and name Data-to-Date Gross-Entrops loss (D2D-CE). The proposed D2D-CE can be expressed as follows:
N
exp (If. Uy - mpl-/T)
LEDDECE =
Σ log
exp (If. Uy; myl-/T) + EjEN() exp (If. f, mul+/7) (6)
N

---

pl;
False negatives mask
Linear Head (P)
Normalization
Feature Extractor (F)
Generator (G)
p(z)
D
X
Advensriial Head
Get mask
Normalizaiion
Get proxy
p(x
Number of samples: |Z|
y
X N
Addv
4D2D-GE CE
Figure 3: Overview of ReACGAN. ReACGAN performs adversarial training using the loss Ladv- At the same time, ReACGAN tries minimize data-to-data cross-entropy loss (D2D-CE) on the linear head (P) to exploit relational information in the labeled dataset. means element- wise product. Note that False negatives mask operates on the similarity matrix between two batches of sample embeddings to compute the similarities between negative samples in the denominator of Eq. (6).
where mp is margin for suppressing a high similarity value between reference sample and its corresponding proxy (easy positive). m, is margin for suppressing low similarity values between negatives samples (easy negatives). [[] and [++ denote min(-.) and max(-.0) functions, respectively,
3.3 Useful Properties of Data-to-Data Cross-Entropy Loss
Sq, G7T
In this subsection, explain four useful proper- ties of D2D-CE. Let 8q be a similarity between the normalized embedding fg and the corresponding normalized proxy Uy,, Sq. be a similarity between f, and fr. and a and be arbitrary indices of nega- tive samples, ie., a, E W()).. Then the properties of D2D-CE be summarized as follows:
Hard negative mining, Easy positive suppression
Hard negative mining
mn
Easy negative suppression
No gradient
Property 1. Hard negative mining. If the value of Sga greater than sg the derivative of LOZD-CE w.rl 8q,a is greater than or equal to the derivative w.rl 8q.i that is &Lmncr Osq.a
mp
Sq = fTVy,
Figure 4: Graph showing regions where hard negative mining, easy positive and negative sup- pression, and no gradient update occur.
2 8Cmpex 2 O.
Property 2. Easy positive suppression. If 8q my 2 0, the derivative of LDZD-CE w.n. Sq is O.
Property 3. Easy negative suppression. If sq. m, 0, the derivative of LozD CE Sg. is O.
Property 4. If $y mp 2 and sg. mn S are satisfied, CD2D-CE has the global minima of N DN log (1 (i)))).
Di
We put proofs of the above properties in Appendix D. Property 1 indicates that D2D-CE implicitly conducts hard negative mining and benefits from comparing samples with each other. Also, Properties 2 and imply that samples will affect gradient updates if the samples are trained sufficiently. Consequently, the classifier concentrates on pushing and pulling hard negative and hard positive examples without being dominated by easy negative and positive samples.
3.4 Rebooted Auxiliary Classifier Generative Adversarial Networks (ReACGAN)
With the proposed D2D-CE objective, we propose the Rebooted Auxiliary Classifier Generative Adversarial Networks (ReACGAN). As ACGAN does. ReACGAN jointly optimizes an adversarial
5

---

loss and the classification objective (D2D-CE). Specifically, the discriminator, which consists of the feature extractor, adversarial head, and linear head, of ReACGAN tries to discriminate whether a given image is sampled from the real distribution not. At the same time, the discriminator tries to maximize similarities between the reference samples and corresponding proxies while minimizing similarities between negative samples using real images and D2DD-EE loss. After updating the discriminator a predetermined number of times, the generator strives to deceive the discriminator by generating well-conditioned images that will output low D2D-CE value. By alternatively training the discriminator and generator until convergence, ReACGAN generates high-quality images of diverse categories without early-training collapse. We attach the algorithm table in Appendix A.
Differences between ReACGAN and ContraGAN. The authors of ContraGAN [23] propose a conditional contrastive loss (2C loss) to cover data-to-data relationships when training cGANs. The main differences between 2C loss and D2D-CE objective are summed up into three points: (1) while 2C loss is derived from NT-Xeet loss [39]. which is popularly used in the field of the self-superviss learning, our D2D-CE is developed to resolve the early-training collapsing problem and the poor generation results of ACGAN, (2) 2C loss holds false-negative samples in the denominator, and they can cause unexpected positive repulsion forces, and (3) 2C loss contains the similarities between all positive samples in the numerator, and they give rise to large gradients on pulling easy positive samples. Consequently, GANs with 2C loss tend to synthesize images of unintended classes as reported in the author's s software document [40]. However, D2D-CE does not contain the false negatives in the denominator and considers only the similarity between a sample and its proxy in the numerator. Therefore, ReACGAN is free from the undesirable repelling forces and does not conduct unnecessary easy positive mining. More detailed explanations are attached in Appendix F.
Consistency Regularization and D2D-CE Loss. Zhang et al. [16] propose a consistency regular- ization to force the discriminator to make consistent predictions if given two images are visually close to each other. They create visually similar image pair by augmenting reference image with pre-defined augmentations. After that, they let the discriminator minimize L2 distance between the logit of the reference image and the logit of the augmented counterpart. While ReACGAN locates an image embedding nearby its corresponding proxy but far apart multiple image embeddings of different classes, consistency regularization only pulls reference and the augmented image towards each other. Since consistency regularization and D2D-CE can be applied together, we will show that ReACGAN benefits from consistency regularization in the experiments section (Table 1).
4 Experiments
4.1 Datasets and Evaluation Metrics
To verify the effectiveness of ReACGAN, we conduct conditional image generation experiments using five datasets: CIFARIO [30], Tiny-ImageNet [32], CUB200 [33], ImageNet [31], and AFHQ [41]
datasets and four evaluation metrics: Inception Score (IS) [42], Fréchet Inception Distance (FID) [34],
and Fo.125 (Precision) and Fy (Recall) [43]. The details on the training datasets are in Appendix C.I.
Inception Score (IS) [42] and Fréchet Inception Distance (FID) [34] are widely used metrics for evaluating generative models. We utilize IS and FID together because some studies [4, 6, 25] have shown that IS has a tendency to measure the fidelity of images better while FID tends to weight
capturing the diversity of images.
Precision (Fo.125) and Recall (Fz) [43] metrics for estimating precision and recall of the approxi- mated distribution p(G(z)) against the true data distribution p(x). Instead of evaluating generative models using one-dimensional scores, such as IS and FID, Sajjadi et al. [43] have suggested using a two-dimensional score Fo.125 and Fg that can quantify how precisely the generated images are and how well the generated images cover the reference distribution.
4.2 Experimental Details
The implementation of ReACGAN basically follows details of SyTorch-StudioGir library [40]] that supports various experimental setups from ACGAN [18] to StyleGAN2 [7] + ADA [8] with
is an open-source library under the MIT license (MIT) with the exception of Style- GAN2 and StyleGAN2 + ADA related implementations, which under the NVIDIA source code license.

---

Table 1: Comparison with classifier- based GANs [18. 23] and projection based GANs [3, 4, 29] on CIFARIO [30], Finy-ImageNet [32], and CUB200 [33] datasets using IS [42], FID [34], Fo. and
F's [43] metrics. For baselines, both the numbers from the cited paper (denoted as * in method) and
from our experiments using StudioGAN library [40] are reported. The numbers in bold-faced denote
the best performance and underline indicate the values are in one standard deviation from the best.
CIFARIO [30]
Ttny-ImageNe (32)
CUB200
(33)
Method
IS 1
FID
Fa 1
Fs f
IS
FID
Fo f F
IS
FID
Fs
SNGAN [3]
8.22
21.7
BigGAN" [4]
9.22
14.73
ContraCiAN [23]
29.49
ACGAN [18]
9.84
8.45
0.993
0.992
6.00
96.04
0.656
0.368
6.09
60.73
0.726
0.891
SNGAN [3]
8.67
13.33
0.985
0.976
8.71
51.15
0.900
0.702
5.41
47.75
0.754
0,912
SAGAN [29]
8.66
14.31
0,983
0.973
8.74
49.90
0.872
0.712
5.48
54.29
0.728
0.882
BigGAN [4]
9.81
8.08
0.993
0.992
12.78
32.03
0.948
0.868
4.98
18.30
0.924
0.967
ContraGAN [23]
9.70
8.22
0.993
0
13.46
28.55
0.974
0.881
5.34
21.16
0.935
0,942
ReACGAN
9.89
7.88
0.994
0.992
14.06
27.10
0.970
0.894
4.91
15.40
0.970
0.954
BigGAN + [16]
BigGAN [4] CR [16]
9.97
7.18
0.995
0,993
15.94
19.96
0.972
0.950
5.14
11.97
0.978
0,981
ContraCiAN [23] + CR [16]
9.59
8.55
0.992
0.972
15.88
19.21
0.983
0.941
4,90
11.08
0.984
0.967
ReACGAN + CR [16]
10.11
2,20
0,996
0.994
16.56
19.69
0.984
0.940
4.87
10.72
0,985
0.971
BigGAN + DiffAug" [9]
9.17
8.49
BigGAN [4] + DiffAug
9.94
7.17
0.995
0.992
18.08
15.70
0.980
0.972
5.53
12.15
0.967
0.981
ContraGiAN (23) DiffAug [9]
9.95
7.27
0.995
0.992
18.20
15.40
0.986
0.963
5.39
11.02
0.978
0.970
ReACGAN DiffAug [9]
10.22
6.79
0.996
0.993
20.60
14.25
0.988
0.972
5.22
9.27
0.985
0.983
Real Data
11.54
34.11
5.49
different scales of datasets [30, 32, 31, 31, and architectures [2, 13, 4, 7]. For fair comparison, we use the same backbones over all baselines, except otherwise noted, for both the discriminator and generator. For stable training, discriminator except for experiments using SNGAN (in this case, we apply SN to the discriminator only) and StyleGAN2 [7]. We also use the same conditioning method for generators with conditional batch normalization (cBN) [44, 45, 19]. This allows us to investigate solely the conditioning methods of the diseriminator, which are the main interest of our paper. If not specified, we use hinge loss [46] as a default for the adversarial loss.
apply spectral normalization (SN) [3] to the generator and
Before conducting main experiments, perform hyperparameter search with candidates of temper- ature r E {0.125, 0.25, 0.5, 0.75, 1.0} and a positive margin my E (0.5, 0.75, 0.9, 0.95, 0.98, 1.0). We set negative margin mn 1 mp to reduce search time. Through extensive ex- periments with 3 runs per each setting, we select т with {0.5, 0.75, 0.25, 0.5, 0.25) and mn with (0.98, 1.0, 0.95, 0.98, 0.90} for CIFARIO, Tiny-ImageNet, CUB200, ImageNet 256 B.S., and ImageNet 2048 B.S. experiments, respectively. A low temperature seems to work well on fine-grained image generation tasks, but generally, ReACGAN is robust to the choice of hyperparameters. The results of the hyperparameter search and other hyperparameter setups are provided in Appendix C.2.
We evaluate all methods through the same protocol of [9. 16. 17], which uses the same amounts of generated images from the reference split specialized for each dataset.? Besides, we run all the experiments three times with random seeds and report the averaged best performances for reliable evaluation with the lone exception of ImageNet and StyleGAN2 related experiments. Please refer to Appendix C.2 for other experimental details. The numbers in bold-faced denote the best performance and in underline indicate that the values are in one standard deviation from the best.
4.3 Evaluation Results
Comparison with Other cGANs. We compare ReACGAN with previous state-of-the-art cGANs in Tables and 2. We employ the implementations of GANs in PyTorch-StudioGAN library as it provides improved results on standard benchmark datasets [30, 31]. For fair comparison, we provide results from each original paper (denoted as * in method) as well as those from StudioGAN
We use the validation split as the default reference set, but we use the test split of CIFAR10 and the training split of CUB200 and AFHQ due to the absence or lack of the validation dataset.
7

---

Table Experiments using ImageNet [31] dataset. B.S. means batch size for training. (Left): Com- parisons with previous cGAN approaches. (Right): Learning curves of BigGAN [4], ContraGAN [23]. and ReACGAN (ours) which are trained using the batch size of 256.
ImngeNet
Method
15 T
FID L
Fasrs 1
F
and FID scores during training
ACGAN"
184.41
SNGAN" [3]
36.80
27.62
SAGAN [29]
52.52
18.28
BigGAAN
BiGGAN
BigGAN [20]
38,05
22.77
CorezaGAN BoACGAN
ContraGAN
TAC-GAN [20]
23.75
FID (lower ia bettter)
ReACGAN
IS (bigher better)
286
ContraGAAN [23]
31.10
19.69
0.951
0.927
SNGAA
ACGAN [18]
62.99
32.25
26.79
26.35
0,935
0.963
14)
SAGAN [29]
29.85
34.73
0.849
0.914
BigGAN [-]
43.97
16.36
0,964
0.955
70-
ContraGAN [23]
25.25
25.16
0.947
0.855
ReACGAN
13.98
BigGAN [4] DiffAug [9]
36.97
18.57
0.956
0.941
50-
ReACGAN DiffAug
69.74
11.95
0,977
0.975
=3= 2018
BigGAN [4]
99.31
8.51
50
BigGAN" [25]
104.57
9.18
ReACGAN
BigGAN [4]
99.71
8:23
7,89
0.991
0.985
0.989
10
1206
14tk 1001:
Real Data
173.33
Iteration
library [40]. We also conduct experiments with popular augmentation- methods: consistency regularization (CR) [16] and differentiable augmentation (DiffAug) [9].
Compared with other cGANs, ReACGAN performs the best most benchmarks, surpassing the previous methods by 2.5%, 5.1%, 15.8%, and 14.5% in FID on CIFARIO, Tiny-ImageNer CUB200, and ImageNet (256 B.S.), respectively. ReACGAN also harmonizes with augmentation-bace regularizations, bringing incremental improvements on all the metrics. For the ImageNet experiments using batch size of 256, ReACGAN reaches higher IS and lower FID with fewer iterations than other models in comparison. Finally, we demonstrate that ReACGAN can learn with larger batch size on ImageNet. While some recent methods [20, 38, 27] have been built on ACGAN improve the generation performance of ACGAN, large-scale image generation experiments with the batch size of 2048 have never been reported. Table shows that our ReACGAN reaches FID score of 8.23 on ImageNet, being comparable with the value of 7.89 from BigGAN implementation in PyTorch-Studine StudioGAAN library. ReACGAN, however, provides better synthesis results than other implementations of BigGAN [4, 25]. Note that our result on ImageNet is obtained in only two runs while the training setup and architecture of BigGAN have been extensively searched and finely tuned.
Comparison with Other Conditioning Losses. We investigate how the generation qualities vary with different conditioning losses while keeping the other configurations fixed. We compare D2D-CE loss with cross cross-entropy loss of ACGAN (AC) [18], loss used in the projection discriminator (PD) [19], multi-hinge loss (MH) [26], and conditional contrastive loss (2C) [23]. The results are shown in Table 3. AC- and MH-based models present decent results on CIFAR 10, but undergo early-training collapse on Tiny- ImageNet and CUB200 datasets. Replacing them with PD, 2C, and D2D-CE losses produce satisfactory performances across all datasets, where PD loss makes the best Fs (recall) on CUB200 dataset while giving third 12.125 (precision) value. The noticeable point is that the proposed D2D-CE shows consistent results across all datasets, showing the lowest FID and the highest Fs and Fo.125 values in most cases. This means ReACGAN can generate high-fidelity images and is relatively free from the precision and recall trade-off [43] than the others.
Consistent Performance of ReACGAN on Adversarial Loss Selection. We validate the consistent performance of ReACGAN on four adversarial losses: non-saturation loss [1]. least square loss [10], Wasserstein loss with gradient penalty regularization (W-GP) [13], and hinge loss [46] on CIFARIO and Tiny-ImageNet datasets in Table 4. The experimental results show that BigGAN + D2D-CE (ReACGAN) consistently outperforms the projection discriminator (PD) and conditional contrastive loss (2C) counterparts over three adversarial losses [1, 13, 46]. However, for the experiments using the least square loss [10], ReACGAN exhibits inferior generation performances to the projection discriminator. We speculate that minimizing the least square distance between an adversarial logit and the target scalar (1 or 0) might affect the norms of feature maps and spoil the classifier training performed by D2D-CE loss.
8

---

Table Experiments on the effectiveness of D2D-CE loss compared with other conditioning losses.
CIFARIO [30]
Tiny ImageNet [32]
CUB200 [33]
Conditioning Method
FID
Fauus 1
Fs 1
IS 1
FID
Fauus 1
Fa
1S
FID
4 Fazs f
F4 f
BigGAN w/o Condition [4]
(Abbreviated Big)
9.46
12.21
0.987
0.982
7.38
76.15
0.804
0.576
5.16
35.17
0.852
0.936
Big AC [18]
9.84
8.45
0.993
0.992
6.00
96.04
0.656
0.368
6.19
60.73
0.726
0.891
Big PD [19]
9.81
8.08
0.993
0.992
12.78
32.03
0.948
0.868
18.30
0.924
0.967
Big MH [26]
10.05
7.94
0.994
0.990
4.37
140:74
0.282
0.156
5.18
245.69
0.625
0.832
Big 2C [C]3
8.22
0.993
0.991
13.46
0.974
0.881
0.942
Big D2D-CE (ReACGAN)
9.89
7.88
0.994
0.992
14,06
27.10
0.970
0.894
4.91
15.40
0.970
0.954
Table 4: Experiments to identify the consistent performance of D2D-CE on adversarial loss selection.
Adversarial Loss
Conditioning Method
CIFARIO [30]
Tiny ImageNet
IS
FID 4
Fi
Better?
FID
Fa
Fg f Better?
PD [19]
9.75
8.29
0,993
0.991
8.27
58.85
0.816
0.713
Non-saturation [1]
2C [23)
9.30
10:47
0.990
0.959
6.57
84.27
0.745
0.556
D2D-CE
9.79
8.27
0.993
0.991
11.76
39.32
0.942
0.852
PD [19]
9.94
8.26
0.993
0.992
12.74
37.14
0.920
0,900
v
Least square [10]
[23]
12.18
0.986
0,941
9.58
0.706
D2D-CE
9.70
9.56
0.991
0.987
9.50
57.67
0.848
0.692
PD [19]
5.71
64.75
0.792
0.652
6.67
84.16
0.696
0.498
W-GP [13]
2C [23]
5.93
55.99
0.842
0.709
6.89
74.45
0.812
0.536
D2D-CE
7,30
35.94
0,942
0.847
8.92
$2.74
0.856
0.689
PD [19]
9.81
8.08
0.993
0,992
12.78
32.03
0.948
0.868
Hinge [46]
2C [23]
9.70
8.22
0.993
0.991
13.46
28.55
0.974
0.881
D2D CE
9.89
7.88
0.994
0.992
14.06
27.10
0.970
0.894
Table 5: FID [34] values of conditional StyleGAN2 (eStyleGAN2) [7] and StyleGAN2 [7] D2D-CE
on CIFAR10 [30] and AFHQ [41] datasets. ADA means the adaptive discriminator augmentation [8]. FID is exceptionally computed using the training split for calculating the reference moments since FID value StyleGAN2 is often calculated using the moments of the training dataset.
Conditioning method
CIFAR1O [30]
AFHQ [41]
cStyleGAN2 [7]
3.88
StyleGAN2 [7] D2D-CE
3.34
cStyleGAN2 [7] ADA [S]
2.43
4.99
StyleGAN2 [7] ADA [8] D2D-CE
2.38
4.95
StyleGAN2 [7] DirtAug [9] D2D-CE Tuning
2.26
Effect of D2D-CE for Different GAN Architectures. We study the effect of D2D-CE loss with different GAN architectures. In Table 5, we validate that D2D-CE is effective for StyleGAN2 [7] backbone and also fits well with the adaptive discriminator augmentation (ADA) [8]. StyleGAN2 with D2D-CE loss produces 13.9% better generation result than the conditional version of StyleGAN2 (cStyleGAN2) on CIFAR 10. Moreover, StyleGAN2 with D2D-CE can be reinforced with ADA or DiffAug when train StyleGAN2 + D2D-CE under the limited data situation. Among GANs, StyleGAN2 DiffAug D2D-CE + Tuning achieves the best performance on CIFARIO, even outperforming some diffusion-based methods [47, 48]. Additional results with other architectures, i.e., a deep convolutional network [2] and a resnet style backbone [13], are provided in Appendix E.
4.4 Ablation Study
We study how each component of ReACGAN affects ACGAN training. By adding or ablating each part of ReACGAN, as shown in Table 6, we identify four major observations. (1) Feature and weight normalization greatly stabilize ACGAN training and improve generation performances on Tiny-ImageNet and CUB200 datasets. (2) D2D-CE enhances the generation performance by considering data-to-data relationships and by performing easy sample suppression (3th and 4th rows). (3) the suppression technique does not work well on the feature normalized cross-entropy loss (5th row). (4) While ACGAN shows high Inception score on ImageNet experiment, it shows relatively

---

AFHQ -112
IMAGENRT 128
CUB200 1282
LLNY-MMAEENT
CIFARIO 322
Figure 5: Curated images generated by the proposed ReACGAN. More qualitative results on ReAC-
GAN, BigGAN [4], ContraGAN [23], and ACGAN [18] are attached Appendix J.
Table 6: Ablation study on feature map normalization, data-to-data consideration, and easy positive and negative sample suppression.
MnnyImmaeNNe Net [32]
CUB200 [33]
ImageNet [49]
Ablation
IS f
FID
Fo.12
Fs
IS f
FID
Fo,12
IS 1
FID |
Fauzs 1
Fg
ACGAN [18]
6.00
96.04
0.656
0.368
6.09
60.73
0.726
0.891
62.99
26.35
0.935
0.963
+ Normulization
13,46
30.33
0.955
0.889
4.78
25.54
0.883
0.952
18.16
36.40
0.879
0.787
+ Data-to-data (Eq. (5))
12.96
28.71
0.967
0.863
5.08
25.12
0.894
0.946
Suppression (Eq. (6))
14.06
27.10
0.970
0.894
15.40
0.954
63.16
0.974
0.974
Data-to-data
12.96
30.79
0.960
0.857
5.39
30.36
0.863
0.947
poor FID, 125, and F3 values compared with the model trained with normalization, data-to-data consideration, and the suppression technique. This result is consistent with the qualitative results on ImageNet, where the images from ACGAN are easily classifiable, but the images from ReACGAN are high quality and diverse. We attribute the improvement to the discriminator that successfully leverages informative data-to-data and data-to-class relationships with casy sample suppression.
Conclusion
In this paper, have analyzed why training ACGAN becomes unstable the number of classes in the dataset increases. By deriving the analytic form of gradient in the classifier and numerically checking the gradient values, exploding problem caused by the unboundedness of input feature vectors and poor classification abil- ity of the classifier in the early training stage. To alleviate the instability and reinforce ACGAN, we have proposed the Data-to-Data Cross-Entropy loss (D2D-CE) and the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN). The experimental results verify the superiority of GANs on five benchmark
have discovered that the unstable training comes from a gradient
ReACGAN compared with the existing classifier- and projection-barte datasets. Moreover, exhaustive analyses on ReACGAN prove that ReACGAN is robust to hyperpa- rameter selection and harmonizes with various architectures and differentiable augmentations.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the IITP grants NN.2001--001196: AI Graduate School Program POSTECH, N0.2021-0-00537: Visual Common Sense, No.2021-0-02068: AI Innovation Hub) funded by Ministry of Science and ICT, Korea.
References
[1] Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances Neural Information Processing Systems (NewrIPS), pages 2672-2680, 2014.
[2] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation I. Learning with Deep Convolutional Generative Adversarial Networks. arXiv preprint arXiv 1511.06434, 2016.
[3] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Gen- erative Adversarial Networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[5] Tero Karras, Samuli Laine, and Timo Aila, A style-bused generator architecture for generative adver- sarial networks. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 1401-4410, 2019.
[6] Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy P. Lillicrap. LOGAN: Latent Optimisation for Generative Adversarial Networks. arXiv preprint arXiv 1912,00953, 2019.
[7] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lchtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 81 10-8119, 2020.
[8] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training Generative Adversarial Networks with Limited Data. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[9] Shengyu Zhao, Zhijian Liu, Ji Lin, Yan Zhu, and Song Han. Diffcrentiable augmentation for data- efficient gan training. arXiv preprint arXiv 10738, 2020.
[10] Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhixiang Wang, and Stephen Paul Smolley. Least Squares Generative Adversarial Networks. In Proceedings of the International Conference Computer Vision (ICCV), pages 2813 2221222,,220 2017.
[11] Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv preprint arXiv 1701.07875. 2017.
[12] Martín Arjovsky and Léon Bottou. Towards Principled Methods for Training Generative Adversarial Networks. In Proceedings of the International Conference Learning Representations (ICLR), 2017.
[13] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved Training of Wasserstein GANs. In Advances in Neural Information Processing Systems (NeurIPS), pages 5767-5777, 2017.
[14] Naveen Kodali, James Hays, Jacob D. Aberntthy, and Zsolt Kira. On Convergence and Stability of GANs. arXiv preprint arXiv 1705.07215, 2018
[15] Zhiming Zhou, Jiadong Liang. Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. Lipschitz generative adversarial nets. In Proceedings of the International Conference on Machine Learning (ICML), pages 7584-7593, 2019.
[16] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lec. Consistency Regularivation for Generative Adversarial Networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[17] Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved Consistency Regularization for GANs. arXiv preprint arXiv 2002.04724, 2020.
11

---

[18] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional Image Synthesis with Auxiliary Classifier GANs. In Proceedings of the International Conference on Machine Learning (ICML), pages 2642-2651, 2017.
[19] Takeru Miyato and Masanori Koyama. cGANs with Projection Discriminator. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[20] Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin Auxilary Classifiers GAN. In Advances in Neural Information Processing Systems (Neur/PS), 2019.
[21] Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening and Coloring Batch Transform for GANs. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[22] Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few- Shot Unsupervised Image-to-Image Translation. In Proceedings of the International Conference on Computer Vision (ICCV). 2019.
[23] Minguk Kang and Jaesik Park. ContraGAN: Contrastive Learning for Conditional Image Generation. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[24] Woohyeon Shim and Minsu Cho. CircleGAN: Generative Adversarial Learning across Spherical Circles. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[25] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian, Omni-GAN: On the Secrets of cGANs and Beyond. arXiv preprint arXiv:2011. 13074, 2020.
[26] Ilya Kavalerov, Wojeiech Czaja, and Rama Chellappa. A multi-class hinge loss for conditional gans. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021.
[27] Liang Hou, Qi Cao, Huawei Shen, and Xueqi Cheng. eGANs with Auxiliary Diseriminative Classifier. arXiv preprint arXiv:2107. 10000 2021.
[28] Han, Ligong and Min, Martin Renqiang and Stathopoulos, Anastasis and Tian, Yu and Gao, Ruijiang and Kadav, Asim and Metaxas, Dimitris N. Dual Projection Generative Adversarial Networks for Conditional Image Generation. In Proceedings of the International Conference Computer Vision (ICCV), 2021.
[29] Han Zhang, Ian Goodfellow, Dimitris Metaxas. and Augustus Odena. Self-Attention Generative Adversarial Networks. Proceedings of the International Conference on Machine Learning (ICML). pages 7354-7363, 2019.
[30] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. PhD thesis, University of Toronto, 2012.
[31] Jin Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei ImageNet: A large-scale hierarchical image database. In Proceedings the IEFF International Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[32] Johnson et al. Tiny ImageNet Visual Recognition Challenge. https://ttin- hhwp-//emnnneeeeeeeuuuaaaa com.
[33] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSI Birds 200. Technical report, California Institute of Technology. 2010.
[34] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Sime-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances Neural Information Processing Systems (NeurIPS), pages 6626-6637, 2017.
[35] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization. In Advances in Neural Information Processing Systems (NeurIPS), pages 271-279 2016.
[36] Akash Srivastava, Lazar Valkov, Chris Russell, Michael Gutmann, and Charles Sutton. VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Leaminng In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[37] Mehdi Mirza and Simon Osindero, Conditional Generative Adversarial Nets. arXiv preprint arXiv 1414.1784, 2014.
12

---

[38] Zhiming Zhou, Han Cai, Shu Rong. Yuxuan Song, Kan Ren, Weinan Zhang, Jun Wang, and Yong Yu. Activation Maximization Generative Adversarial Nets, In Proceedings of the International Conference on Learning Reprexentations (ICLR), 2018.
[39] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A Simple Framework for Contrastive Learning Visual Representations. arXiv preprint arXiv 2002.05709, 2020.
[40] Minguk Kang and Jaesik Park. Pytorch-studiogan. httpp:///gtthb. cot.://TECC CVLab/
SSSdiiiGAA,, 2020.
[41] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. StarGAN v2: Diverse Image Synthesis for Multiple Domains. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Jung-Won
[42] Tim Salimans, Goodfellow, Wajeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Chen. Im- proved Techniques for Training GANs. In Advances in Neural Information Processing Systems (NeurIPS), pages 2242-2 2016.
[43] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[44] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. Learned Representation For Artistic Style. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
[45] Harm de Vries, Florian Strub, Jeremie Mary. Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. Advances Neural Information Processing Systems (NeurlPS), pages 6594-6604, 2017.
[46] Jac Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv preprint arXiv 1705.02894, 2017.
[47] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the International Conference on Machine Learning (ICML). 2021.
[48] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Score Matching Model Unbounded Data Score. arXiv preprint 2rXiv:2106..0522, 2021.
2021.
[49] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos 7afeirion. Areface: Additive angular margin loss deep face recognition. In Proceedings of the IEFE International Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
[Yes] The main contributions of our paper can be summarized into two points: (1) analyzing why ACGAN training becomes unstable the number of classes in the training dataset increases, (2) developing a new cGAN model: the Rebooted Auxil- iary Classifier Generative Adversarial Networks (ReACGAN), Both claims are well reflected in the abstract and introduction.
(b) Did you describe the limitations of your work?
[Yes] The proposed ReACGAN generates high-quality images using informative data- to-data and data-to-class relationships in the dataset. However, it requires additional hyperparameter tuning for a temperature and margins values. Also, we analyze the weakness of ReACGAN in Appendix F.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Generative Adversarial Networks can be used to synthesize indistinguishable images from the real. This might cause fake news, sexual harnesses, and hacking machine vision applications. Although the potential negative social impacts are not discussed in our main paper, we deal with this issue in Appendix G.
(d) Have you read the ethics review and ensured that your paper conforms to them? [Yes] We carefully read the ethics review guidelines and comply with them.
13

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results?
[Yes] We state all assumptions that arise from the theoretical results, i.e., Property 1-4 Sec. 3.3 in the main paper.
(b) Did you include complete proofs of all theoretical results?
[Yes] We provide the complete proofs in Appendix D.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)?
[Yes] We release model weights and the software package used in paper at www.o//SOOTTCCLLO//////.. Torch-StudioGiAN.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We state details for training models Sec. 4.2 in the main paper, which include the evaluation procedure, hyperparameter selection, and controlled environment for a fair comparison. In addition, we add how we pre-process training datasets in the Appendix C.I.
(c) Did you report
bars (e.g., with respect the random seed after running experi- ments multiple times)?
all the experiments three times with random seeds. We provide standard deviations of Tables 1, 3, 4, Appendix I.
[Yes] For reliable evaluation,
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)?
[Yes] We include the total amount compute and the type of resources Appendix H.
4. If you using existing assets (e-g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite papers and GitHub repository that we used for our experiments in Sec. 4.2.
(b) Did you mention the license of the assets?
[No] We utilize five datasets validate our method: CIFARIO, Tiny-ImageNer CUB200, ImageNet, and AFHQ. These are widely used public datasets, and we cite them in the paper. However, we could not find any license information for CIFAR 10, and CUB200. The ImageNet and AFHQ datasets can be used for non- commercial purposes only. addition, we mention the license of Py TorchhSttuiiodee in the main paper, which is used for our experiments.
(c) Did you include any new assets either in the supplemental material or as a URL? Yes] We provide the URL of our source code the abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating?
[Yes] The CIFAR10. TingeImag,Net, CUB200, ImageNet, and AFHQ datasets are properly curated, and they are popular datasets for research purpose. To our best knowledge, there no inappropriate contents in the datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] The benchmark datasets we used are widely adopted by others and do contain any harmful or troublesome contents.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] Our research has no relevance to the crowdsourcing human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable?
[N/A] Our research has relevance to the crowdsourcing human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] Our research has no relevance to the crowdsourcing or human subjects.
14


Reviews:
[{"review_id": "wexyTwG5xP4", "reviewer": "Reviewer_mphj", "summary": "This paper proposes ReACGAN which is an improved version of ACGAN. The proposed method addresses the gradient exploding of ACGAN in the early training stage by normalizing the feature embeddings onto a unit hypersphere. The authors propose D2D-CE, which is analogous to contrastive loss, to effectively utilize relational information in the dataset. In addition, the authors introduce thresholds to focus on pushing and pulling hard negative and positive samples, further enhancing the conditional generation capability of ACGAN.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written and the authors well investigated the problem of ACGAN and the  proposed method both theoretically and empirically. However, my concern is the novelty of the proposed method because most of the techniques here have been explored in other works (D2D-CE is similar to [1], and the motivation of the thresholds is similar to [2].).\n\n[1] ContraGAN: Contrastive Learning for Conditional Image Generation, NeurIPS 2020.\n\n[2] Focal Loss for Dense Object Detection, ICCV 2017.\n\n\nI have a minor question: MHGAN paper reported IS and FID on Imagenet dataset. Was there any reason not to include this result in Table 2 of the paper?\n\n\nDespite the concerns, the authors showed strong experimental results, and I look forward to acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "gjjQNnir0q", "reviewer": "Reviewer_1hzt", "summary": "This paper proposes to revisit the AC-GAN framework, which usually suffers from training instability, mode collapse and lack of diversity. The authors identify two limitations and propose solutions. First, the classifier causes exploding gradients which is addressed by normalizing feature embeddings. Second, the authors replace the softmax classifier by a contrastive classifier. The experimental results show consistent improvements in the common metrics.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nWhile AC-GAN, L2 normalized features and contrastive learning are certainly not novel and well-known, the study of the AC-GAN gradient problem is somewhat novel. The authors propose an effective method to address the problem and a modified contrastive loss showing certain improvement over ContraGAN. It can be seen as an effective combination of AC-GAN and ContraGAN.\n\nQuality:\nThe submission is technically sound, with extensive theoretical and experimental analysis. The authors explained in detail the relation and differences with closely related works.\n\nClarity:\nThe submission writing is very clear and well organized.\n\nSignificance:\nThe results are significant, especially in Imagenet, which is the most challenging setting. The conclusions are solid and provide valuable insight on the interplay between classifier and GAN.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is not any section about limitations, but the authors have included a section in the supplementary describing potential misuses of GANs and total training time.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "bHSdo9wkUQh", "reviewer": "Reviewer_JFMx", "summary": "The paper proposes approaches to reboot GANs with auxiliary classifier: a) normalizing the classifier layer and b) a data-to-data cross-entropy loss that encourages the auxiliary classifier to learn different class embeddings for different instances. Experiments demonstrate state-of-the-art class-conditional generation performance on CIFAR10, CUB200, and simplified ImageNet datasets with the specified backbone and better performance when working with CR and DiffAugment.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed techniques are well-motivated, sound novel and reasonable.\n- With the proposed techniques, the paper successfully brings auxiliary classifier, an alternative that was deemed to be surpassed by the projection-based approach, into state-of-the-art cGANs, which may invoke new thoughts about auxiliary classifier in the domain.\n\nWeaknesses:\n- The experiments are limited to BigGAN related backbones. The readers would also be interested about whether the proposed techniques can help StyleGANs for conditional generation.\n- I think deeper analysis is needed to fully understand the proposed techniques. I am curious about why the proposed normalization technique can help even when Spectral Normalization is used. Is the last classifier layer using SN? I think that the gradient issue as the authors discovered may not be the root cause. Is it related to discriminator overfitting? Can we alleviate the issue by gradient clipping or lowering the weight of the auxiliary cross-entropy loss?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Good", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ZYi32GyDYVG", "reviewer": "Reviewer_KmwA", "summary": "This paper firstly identifies that the main reason for the GAN's training instability is gradient exploding, and normalize the discriminator's feature to avoid it. Then authors propose a D2D-CE loss to improve the generator model's generalization performance. The extensive experiments including ablation study show the effectiveness and superiority of the proposed method over baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages:\n   1. The presentation is clear and thus it is easy to follow the main idea.\n   2. The proposed method is simple and well-motivated.\n   3. Authors design extensive experiments in the main paper and supplementary material to evaluate the effectiveness of proposed methods, and the results are convincing.\n\nDisadvantages:\n   1. D2D-Loss needs to compute the similarity of Data-to-data in the same mini-batch and the similarity with their proxy, which may be time-consuming. It is better to report and compare the training time/device of the proposed method and baselines.\n   2. I notice that the proposed method perform poorly on the Imagenet classification as Table A9 shows. One main reason is that the evaluation method can only evaluate the quality instead of the diversity of generated images. An alternative is training a classifier on the generated images and tests its accuracy on the real test set to evaluate both the quality and diversity of generated images. Specifically, the accuracies of the classifier may be unstable in the training procedure because of the randomness of generated images, and thus authors can report the best accuracies and std of each training procedure.\n  3.  If the authors have enough time, it will be better to perform synthetic experiments in TAC-GAN, which is a simple way to evaluate whether the proposed methods can fit the true data distribution or not.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the section \"Main Review\"", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review:  This paper proposes two improvements to address the low diversity problem of auxiliary classifiers GANs. First, the classifier is projected to a unit hypersphere to avoid early training collapse caused by gradient exploding. Second, a data to data cross entropy loss, which is similar to the contrastive gan loss, is used to better explore class information. Though the proposed method is rather heuristic, it shows impressive results in conditional image generation on real natural images. The reviewers agree that the proposed method is technically sound, novel, and the paper is well written and organized. However, as pointed out by reviewer KmwA, the proposed method does not mitigate the theoretical problem of ACGAN when the class distributions have support overlaps, which is also verified on the additional experiments on the simulated data.  I would recommend acceptance of this paper given its novelty and impressive performance, and I highly suggest the authors add simulations as done in the TAC paper and report the results of the combination of TAC in their real experiments (with proper discussions), as suggested by reviewer KmwA.
Decision: Accept (Poster)