Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training
Authors: Minguk Kang
Woohyeon Joseph Shim
Minsu Cho
Jaesik Park
Venue: NeurIPS 2021 Poster (2021)
Keywords: Generative Adversarial Networks
Conditional Image Generation
Adversarial Learning
Image Synthesis

Abstract: Conditional Generative Adversarial Networks (cGAN) generate realistic images by incorporating class information into GAN. While one of the most popular cGANs is an auxiliary classifier GAN with softmax cross-entropy loss (ACGAN), it is widely known that training ACGAN is challenging as the number of classes in the dataset increases. ACGAN also tends to generate easily classifiable samples with a lack of diversity. In this paper, we introduce two cures for ACGAN. First, we identify that gradient exploding in the classifier can cause an undesirable collapse in early training, and projecting input vectors onto a unit hypersphere can resolve the problem. Second, we propose the Data-to-Data Cross-Entropy loss (D2D-CE) to exploit relational information in the class-labeled dataset. On this foundation, we propose the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN). The experimental results show that ReACGAN achieves state-of-the-art generation results on CIFAR10, Tiny-ImageNet, CUB200, and ImageNet datasets. We also verify that ReACGAN benefits from differentiable augmentations and that D2D-CE harmonizes with StyleGAN2 architecture. Model weights and a software package that provides implementations of representative cGANs and all experiments in our paper are available at https://github.com/POSTECH-CVLab/PyTorch-StudioGAN.

Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training
Minguk Kang
Woohyeon Shim
Minsu Cho
Jaesik Park
Pohang University of Science and Technology (POSTECH), South Korea {mgkang, wh. shim, mscho, beesik.parke @postech. partskaeeeerp ac.kr
Abstract
Conditional Generative Adversarial Networks (cGAN) generate realistic images by incorporating class information into GAN. While one of the most popular cGANs is an auxiliary classifier GAN with softmax cross-entropy loss (ACGAN), it is widely known that training ACGAN is challenging as the number of classes in the dataset increases. ACGAN also tends to generate easily classifiable samples with lack of diversity. In this paper, we introduce two cures for ACGAN. First, we identify that gradient exploding in the classifier can cause an undesirable collapse in early train- ing, and projecting input vectors onto a unit hypersphere can resolve the problem. Second, we propose the Data-to-Data Cross-Entropy loss (D2D-CE) to exploit propose relational information in the class-labeled dataset. On this foundation, the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN). The experimental results show that ReACGAN achieves state-of-the-a -ftt----t---tttt generation results on CIFAR1O 10, Tiny- ImageNet, CUB200, and ImageNet datasets. We also verify that ReACGAN benefits from differentiable augmentations and that D2D-CE harmonizes with StyleGAN2 architecture. Model weights and a software package that provides implementations of representative eGANs and all experiments in paper are available at
Torch-StudioGAN,
Introduction
Generative Adversarial Networks (GAN) [1] are known the forefront approach to generating high-fidelity images of diverse categories [2, 3, 4, 6, 7, 8, 9]. Behind the sensational generation ability of GANs, there has been tremendous effort to develop adversarial objectives free from the vanishing gradient problem [10, 11, 12], regularizations for stabilizing adversarial training [11, 13, 14, 3. 15, 16, 17], and conditioning techniques to support the adversarial training using category information of the dataset [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28). Subsequently, the conditioning techniques have become the de facto standard for high-quality image generation. The models with the conditioning methods are called conditional Generative Adversarial Networks (cGAN), and cGANs can be divided into two groups depending on the discriminator's conditioning way: classifier-based GANs [18, 20, 23, 25, 27] and projection-based GANs [19, 3, 28].
The classifier-based GANs facilitate an auxiliary classifier to generate class-specific images by penalizing the generator if the synthesized images are not consistent with the conditioned labels. ACGAN [18] has been one the widely used classifier- based GANs for its simple design and satisfactory generation performance. While ACGAN and pulling classifier's s weights (proxies) against image embeddings [23]. it is well known that ACGAN training is prone to collapsing the early stage of training the number of classes increases [19, 23, 25, 27]. In addition, the generator of ACGAN tends to generate easily classifiable images the cost of reduced diversity [18, 19, 27]. Projection- bbased GANs, the other hand, have shown cutting-edge generation results on datasets with a large number of categories. SNGAN [3],
exploit class information by pushing
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Rd
Ra
Positive
Whird *
Wiird
S4
ddg
dog
w dog
Positive
Easy negative
#lowerr
W flower
mp
My
(a) ACGAN [18]
(c) MHGAN [26]
R
P dog
g+
Reference
ddog
Negative
Hard negative
Projection-based
(d) ContraGAN [23)
(e) ReACGAN (Ours)
GAN [19]
Figure 1: Schematics that depict how cGANS perform conditioning. The red color means positive samples/proxy, and the blue color indicates negative samples/proxies. Arrows represent push-pul forces based on the reference sample. The length of an arrow indicates the magnitude of the force.
SAGAN [29], and BigGAN [4] are representatives in this family and can generate realistic images on CIFARIO 10 [30] and ImageNet [31] datasets. However, projection-based GANs only consider pairwise relationship between an image and its label proxy (data-to-class relationships). As the result, the projection-basee based GANs can miss an additional opportunity consider relation information between data instances (data-to-data relationships) as discovered by [23].
In this paper, analyze why ACGAN training becomes unstable the number of classes increases and propose remedies for (1) the instability and (2) the relatively poor generation performance of ACGAN compared with the projection- based models. First, begin by analytically deriving the gradient of the softmax cross- cross-entropy loss used in ACGAN. By examining the exact values of analytic gradients, discover that the unboundedness of input feature vectors and poor classification performance in the early training stage can cause an undesirable gradient exploding problem. Second, propose the Rebooted Auxiliary Classifier Generative Adversarial
with alleviating the instability, Networks (ReACGAN) using the Data-to-Data Cross-Entropy loss (D2D-CE). ReACGAN projects image embeddings and proxies onto unit hypersphere and computes similarities for data-to-data and data-to-class consideration. Additionally, we introduce two margin values for intra-class variations and inter-class separability. In this way, ReACGAN overcomes the training instability and can exploit additional supervisory signals by explicitly considering data-to-clate and data-to aata-to-data relationships, and also by implicitly looking at class-to-class relationships in the same
To validate our model, we conduct image generation experiments on CIFAR1O [30]. Tiny- ImageNet [32], CUB200 [33], and ImageNet [31] datasets. Through extensive experiments, demonstrate that ReACGAN beats both the classifier- based and projection-based GANs, improving the state of the art by 2.5%, 15.8%, 5.1%, and 14.5% in terms of Fréchet Inception Dis- tance (FID) [34] on the four datasets, respectively. We also verify that ReACGAN benefits from consistency regularization [16] and differentiable augmentations [9, 8] for limited data training. Finally, we confirm that D2D-CE harmonizes with the StyleGAN2 architecture [7].
Background: Generative Adversarial Networks
Generative Adversarial Network (GAN) [1] is an implicit generative model that aims to generate a
sample indistinguishable from the real. GAN consists of two networks: Generator G Z X
that tries to map a latent variable z ~ p(z) into the real data space X' and a Diseriminator D X -
[0, that strives to discriminate whether given sample is from the real data distribution Preat (x) from the implicit distribution Pgen (G(z) derived from the generator G(z). The objective a vanilla
GAN [1] can be expressed follows:
min max Ex-mp(x) (lw.ol/ooooooooo [Dx))) + En- (og -D G D
(1)

---

While GANs have shown impressive results in the image generation task [2. 35, 11, 13], training GANs often ends up encountering mode-collapse problem [36, 11 12]. As one of the preseriptions for stabilizing and reinforcing GANs, training GANs with categorical information, named conditional Generative Adversarial Networks (cGAN), is suggested [37, 18, 19]. Depending on the presence of explicit classification losses, cGAN can be divided into two groups: Classifier- GANs [18, 20, 23, 25, 27] and Projection-base GANs [19, 3, 4, 28]. One of the widely used classifier-based GANs is ACGAN [18], and ACGAN utilizes softmax cross-entropy loss to perform classification task with adversarial training. Although ACGAN has shown satisfactory generation results, training ACGAN becomes unstable as the number of classes in the training dataset increases [19, 23, 25, 27]. Besides, ACGAN tends to generate easily classifiable images at the cost of limited diversity [18. 19, 27]. To alleviate those problems, Zhou et al. [38] have proposed performing adversarial training on the classifier. Gong el al. [20] have introduced an additional classifier to eliminate a conditional entropy minimization process in the adversarial training. However, ACGAN training still suffers from the early-training collapse issue and the reduced diversity problem when trained on datasets with a large [32] and ImageNet [31].
number of class categories, such
In these circumstances, Miyato et al. [19] have proposed a projection discriminator for cGANs and have shown significant improvement in generating the ImageNet dataset. Motivated by the promising result of the projection discriminator, many projection-basee GANs [3. 29, 4, 16, 6, 17] have been proposed and become the standard for conditional image generation. In this paper, revisit ACGAN and unveil why ACGAN training is so unstable. Coping with the instability, we propose the Rebooted Auxiliary Classifier GANs (ReACGAN) for high-quality and diverse image generation.
3 Rebooting Auxiliary Classifier GANs
3.1 Feature Normalization
To uncover a nuisance that can cause the instability of ACGAN, we start by analytically deriving the gradients of weight vectors in the softmax classifier. Let the part the discriminator before the fully connected layer be a Feature extractor F X - F E ı2d and let Classifier C F - R be a single fully connected layer parameterized by W = [W].... wel E Raxe, where e denotes the number of classess. We sample training images X = {œ1, ..... NN } and integer labels = {yI..... } from the joint distribution p(x,y) Using the notations above, we can express the empirical cross-entropy loss used in ACGAN [18] as follows:
1 N LCE = - N
exp (F(x:) wy,
Σ log
Σラー exp (T(::))T)) Two,)
Based on Eq. (3), we can derive the derivative of the cross-entropy loss, w.r.t (1....... as follows:
OCcr 1 N
Σ 1=1
F(a.)
11 Pitt)
Ow:
N
where 1y is an indicator function that will output 1 if ys = is satisfied, and pt.k is a class probability that represents the probability that i-th sample belongs to class k. mathematically (xp,)( mnx
>e= (x. w The equation above implies that the norm of the gradient of the softmax cross-entropy loss is coupled with the norms and directions of each input feature map F(x;)) and the class probabilities. In the early training stage, the classifier is prone to making incorrect predictions, resulting in low probabilities. This phenomenon occurs more frequently as the number of categories in the dataset increases. As the result, the norm of the gradient Pwry begins to explode as the vector F(x;) stretches out to we direction but being located close to the the other vectors (1( This often breaks the balance between adversarial learning and classifier training, leading to an early-training collapse. Once the early-training collapse occurs, ACGAN training concentrates on classifying categories of images instead of discriminating the authenticity of given samples. We experimentally demonstrate that the average norm of ACGAN's input feature maps increases as the training progresses (Fig. 2a). Accordingly, the average norm of the gradients increases sharply at the early training stage and decreases with the high class probabilities of the classifier (Fig. 2b, Fig. A3 Appendix). While the average norm of gradients decreases at some point, the FID value of ACGAN does not decrease, implying the collapse of ACGAN training (Fig. 2c).
@Ccc
3

---

Average nomes of impat frature maaee
Aversay norms of grodions of classilie
Thends of FID scvess
FID (loner in bester))
Norn of fouture
Norre of gralltee
A01-1
Schmei Namoalle
61
DMD Seggreccon
D2D-CE
0.30
16
0.301
Bletation
Iteration
Revniionn
(a) Feature norm
(b) Gradient norm
(c) FID score
Figure 2: Merits of integrating feature normalization and data-to-data relationship consideration. All experiments are conducted on Tiny-ImageNer [32] dataset. (a) Average norms of input feature maps F(-), (b) average norms of gradients of classification losses, and (c) trends of FID scores. Compared with ACGAN [18], the proposed ReACGAN does not experience the training collapse problem caused by excessively large norms of feature maps and gradients at the early training stage. In addition, ReACGAN can converge to better equilibrium by considering data-to-data relationships with easy positive and negative sample suppression.
As one of the remedies for the gradient exploding problem, we find that simply normalizing the F(a.)
feature embeddings onlo unil hypersphere [[[(,,]]] effectively resolves ACGAN's early-training collapse (see Fig. 2). The motivation is that normalizing the features onto the hypersphere makes the norms of feature maps equal to 1.0. Thus, the discriminator does not experience the gradient exploding problem. From the next section, we will deploy a linear projection layer P on the feature extractor F. And, we will normalize both the embeddings from the projection layer and the weight vectors (w),... ..., w.) in the classifier since normalizing both the embeddings and the weight vectors does not degrade image generation performance. We denote the normalized embedding P(F(2.)) [P(F(,))]]
as fü and the normalized weight vector ||,,
Wi as Byy-
3.2 Data-to-Data Cross-Entropy Loss (D2D-CE)
We expand the feature normalized softmax cross-entropy loss described in Sec. 3.1 the Data-to-Data Cross-Entropy (D2D-CE) loss. The motivations are summarized into two points: (1) replacing data- to-class similarities in the denominator of Eq. (3) with data-to-data similarities and (2) introducing two margin values to the modified softmax cross-entropy loss. We expect that point (1) will encourage the feature extractor consider data-to-class as well as data-to-data relationships, and that point (2) will guarantee inter-class separability and intra-class variations in the feature space while preventing ineffective gradient updates induced by easy negative and positive samples. To develop the feature normalized cross- cross-entrops entropy loss into D2D-CE, we replace the similarities between a sample embedding
and all proxies except for the positive one in the denominator, (). ....)) exp (ff, 0y) with similarities between negative samples in the same mini-batch. The modified cross-entropy loss can be expressed as follows:
N Σ 2=1
exp (f, Uy./T) exp (f EjeN(o exp (f. (/T)))
fCF N log
(5)
where 7 is a temperature, and (i)) is the set of indices that point locations of the negative samples whose labels are different from the reference label vy, the mini-batch. The self semilaimmlarity matrix of samples in the mini-batch used calculate the similarities between negative samples f. fjeN() with a false negative mask (see Fig. 3). Thus, Eq. (5) enables the discriminator to contrastively compare visual differences between multiple images and can supply more informative supervision for image conditioning. Finally, we introduce two margin hyperparameters to E and name Data-to-Dala Gross-Entrops loss (D2D-CE). The proposed D2D-CE can be expressed as follows:
N
exp (If. Uy - mpl-/T)
LEDDECE =
Σ log
exp (If. Uy; myl-/T) + EjEN() exp (If. f, mul+/7) (6)
N

---

pl;
False negatives mask
Linear Head (P)
Normalization
Feature Extractor (F)
Generator (G)
p(z)
D
X
Advensriial Head
Get mask
Normalizaiion
Get proxy
p(x
Number of samples: |Z|
y
X N
Addv
4D2D-GE CE
Figure 3: Overview of ReACGAN. ReACGAN performs adversarial training using the loss Ladv- At the same time, ReACGAN tries minimize data-to-data cross-entropy loss (D2D-CE) on the linear head (P) to exploit relational information in the labeled dataset. means element- wise product. Note that False negatives mask operates on the similarity matrix between two batches of sample embeddings to compute the similarities between negative samples in the denominator of Eq. (6).
where mp is margin for suppressing a high similarity value between reference sample and its corresponding proxy (easy positive). m, is margin for suppressing low similarity values between negatives samples (easy negatives). [[] and [++ denote min(-.) and max(-.0) functions, respectively,
3.3 Useful Properties of Data-to-Data Cross-Entropy Loss
Sq, G7T
In this subsection, explain four useful proper- ties of D2D-CE. Let 8q be a similarity between the normalized embedding fg and the corresponding normalized proxy Uy,, Sq. be a similarity between f, and fr. and a and be arbitrary indices of nega- tive samples, ie., a, E W(().. Then the properties of D2D-CE be summarized as follows:
Hard negative mining, Easy positive suppression
Hard negative mining
mn
Easy negative suppression
No gradient
Property 1. Hard negative mining. If the value of Sga greater than sg the derivative of LOZD-CE w.rl 8q,a is greater than or equal to the derivative w.El 8q.i that is &Lmncr dsq.a
mp
Sq = fTVy,
Figure 4: Graph showing regions where hard negative mining, easy positive and negative sup- pression, and no gradient update occur.
2 8Cmpex 2 O.
Property 2. Easy positive suppression. If 8q my 2 0, the derivative of LDZD-CE w.n. Sq is O.
Property 3. Easy negative suppression. If sq. m, 0, the derivative of LozD CE Sg. is O.
Property 4. If $y mp 2 and sg. mn S are satisfied, CD2D-CE has the global minima of N DN log (1 (i)))).
Di
We put proofs of the above properties in Appendix D. Property 1 indicates that D2D-CE implicitly conducts hard negative mining and benefits from comparing samples with each other. Also, Properties 2 and imply that samples will affect gradient updates if the samples are trained sufficiently. Consequently, the classifier concentrates on pushing and pulling hard negative and hard positive examples without being dominated by easy negative and positive samples.
3.4 Rebooted Auxiliary Classifier Generative Adversarial Networks (ReACGAN)
With the proposed D2D-CE objective, we propose the Rebooted Auxiliary Classifier Generative Adversarial Networks (ReACGAN). As ACGAN does. ReACGAN jointly optimizes an adversarial
5

---

loss and the classification objective (D2D-CE). Specifically, the discriminator, which consists of the feature extractor, adversarial head, and linear head, of ReACGAN tries to discriminate whether a given image is sampled from the real distribution not. At the same time, the discriminator tries to maximize similarities between the reference samples and corresponding proxies while minimizing similarities between negative samples using real images and D2DD-EE loss. After updating the discriminator a predetermined number of times, the generator strives to deceive the discriminator by generating well-conditioned images that will output low D2D-CE value. By alternatively training the discriminator and generator until convergence, ReACGAN generates high-quality images of diverse categories without early-training collapse. We attach the algorithm table in Appendix A.
Differences between ReACGAN and ContraGAN. The authors of ContraGAN [23] propose a conditional contrastive loss (2C loss) to cover data-to-data relationships when training cGANs. The main differences between 2C loss and D2D-CE objective are summed up into three points: (1) while 2C loss is derived from NT-Xeet loss [39]. which is popularly used in the field of the self- learning, our D2D-CE is developed to resolve the early-training collapsing problem and the poor generation results of ACGAN, (2) 2C loss holds false-negative samples in the denominator, and they can cause unexpected positive repulsion forces, and (3) 2C loss contains the similarities between all positive samples in the numerator, and they give rise to large gradients on pulling easy positive samples. Consequently, GANs with 2C loss tend to synthesize images of unintended classes as reported in the author's s software document [40]. However, D2D-CE does not contain the false negatives in the denominator and considers only the similarity between a sample and its proxy in the numerator. Therefore, ReACGAN is free from the undesirable repelling forces and does not conduct unnecessary easy positive mining. More detailed explanations are attached in Appendix F.
Consistency Regularization and D2D-CE Loss. Zhang et al. [16] propose a consistency regular- ization to force the discriminator to make consistent predictions if given two images are visually close to each other. They create visually similar image pair by augmenting reference image with pre-defined augmentations. After that, they let the discriminator minimize L2 distance between the logit of the reference image and the logit of the augmented counterpart. While ReACGAN locates an image embedding nearby its corresponding proxy but far apart multiple image embeddings of different classes, consistency regularization only pulls reference and the augmented image towards each other. Since consistency regularization and D2D-CE can be applied together, we will show that ReACGAN benefits from consistency regularization in the experiments section (Table 1).
4 Experiments
4.1 Datasets and Evaluation Metrics
To verify the effectiveness of ReACGAN, we conduct conditional image generation experiments using five datasets: CIFARIO [30], Tiny-ImageNet [32], CUB200 [33], ImageNet [31], and AFHQ [41]
datasets and four evaluation metrics: Inception Score (IS) [42], Fréchet Inception Distance (FID) [34],
and Fo.125 (Precision) and Fy (Recall) [43]. The details on the training datasets are in Appendix C.I.
Inception Score (IS) [42] and Fréchet Inception Distance (FID) [34] are widely used metrics for evaluating generative models. We utilize IS and FID together because some studies [4, 6, 25] have shown that IS has a tendency to measure the fidelity of images better while FID tends to weight
capturing the diversity of images.
Precision (Fo.125) and Recall (Fz) [43] metrics for estimating precision and recall of the approxi- mated distribution p(G(z)) against the true data distribution p(x). Instead of evaluating generative models using one-dimensional scores, such as IS and FID, Sajjadi et al. [43] have suggested using a two-dimensional score Fo.125 and Fg that can quantify how precisely the generated images are and how well the generated images cover the reference distribution.
4.2 Experimental Details
The implementation of ReACGAN basically follows details of SyTorch-StudioGir library [40]] that supports various experimental setups from ACGAN [18] to StyleGAN2 [7] + ADA [8] with
is an open-source library under the MIT license (MIT) with the exception of Style- GAN2 and StyleGAN2 + ADA related implementations, which under the NVIDIA source code license.

---

Table 1: Comparison with classifier- based GANs [18. 23] and projection based GANs [3, 4, 29] on CIFARIO [30], Finy-ImageNet [32], and CUB200 [33] datasets using IS [42], FID [34], Fo. and
F's [43] metrics. For baselines, both the numbers from the cited paper (denoted as * in method) and
from our experiments using StudioGAN library [40] are reported. The numbers in bold-faced denote
the best performance and underline indicate the values are in one standard deviation from the best.
CIFARIO [30]
Tuy-Imageei (32)
CUB200
(33)
Method
IS 1
FID
Fa 1
Fs f
IS
FID
Fo f F
IS
FID
Fs
SNGAN [3]
8.22
21.7
BigGAN" [4]
9.22
14.73
ContraCiAN [23]
29.49
ACGAN [18]
9.84
8.45
0.993
0.992
6.00
96.04
0.656
0.368
6.09
60.73
0.726
0.891
SNGAN [3]
8.67
13.33
0.985
0.976
8.71
51.15
0.900
0.702
5.41
47.75
0.754
0,912
SAGAN [29]
8.66
14.31
0,983
0.973
8.74
49.90
0.872
0.712
5.48
54.29
0.728
0.882
BigGAN [4]
9.81
8.08
0.993
0.992
12.78
32.03
0.948
0.868
4.98
18.30
0.924
0.967
ContraGAN [23]
9.70
8.22
0.993
0
13.46
28.55
0.974
0.881
5.34
21.16
0.935
0,942
ReACGAN
9.89
7.88
0.994
0.992
14.06
27.10
0.970
0.894
4.91
15.40
0.970
0.954
BigGAN + [16]
BigGAN [4] CR [16]
9.97
7.18
0.995
0,993
15.94
19.96
0.972
0.950
5.14
11.97
0.978
0,981
ContraCiAN [23] + CR [16]
9.59
8.55
0.992
0.972
15.88
19.21
0.983
0.941
4,90
11.08
0.984
0.967
ReACGAN + CR [16]
10.11
2,20
0,996
0.994
16.56
19.69
0.984
0.940
4.87
10.72
0,985
0.971
BigGAN + DiffAug" [9]
9.17
8.49
BigGAN [4] + DiffAug
9.94
7.17
0.995
0.992
18.08
15.70
0.980
0.972
5.53
12.15
0.967
0.981
ContraGiAN (23) DiffAug [9]
9.95
7.27
0.995
0.992
18.20
15.40
0.986
0.963
5.39
11.02
0.978
0.970
ReACGAN DiffAug [9]
10.22
6.79
0.996
0.993
20.60
14.25
0.988
0.972
5.22
9.27
0.985
0.983
Real Data
11.54
34.11
5.49
different scales of datasets [30, 32, 31, 31, and architectures [2, 13, 4, 7]. For fair comparison, we use the same backbones over all baselines, except otherwise noted, for both the discriminator and generator. For stable training, discriminator except for experiments using SNGAN (in this case, we apply SN to the discriminator only) and StyleGAN2 [7]. We also use the same conditioning method for generators with conditional batch normalization (cBN) [44, 45, 19]. This allows us to investigate solely the conditioning methods of the diseriminator, which are the main interest of our paper. If not specified, we use hinge loss [46] as a default for the adversarial loss.
apply spectral normalization (SN) [3] to the generator and
Before conducting main experiments, perform hyperparameter search with candidates of temper- ature r E {0.125, 0.25, 0.5, 0.75, 1.0} and a positive margin my E (0.5, 0.75, 0.9, 0.95, 0.98, 1.0). We set negative margin mn 1 mp to reduce search time. Through extensive ex- periments with 3 runs per each setting, we select т with {0.5, 0.75, 0.25, 0.5, 0.25) and mn with (0.98, 1.0, 0.95, 0.98, 0.90} for CIFARIO, Tiny-ImageNet, CUB200, ImageNet 256 B.S., and ImageNet 2048 B.S. experiments, respectively. A low temperature seems to work well on fine-grained image generation tasks, but generally, ReACGAN is robust to the choice of hyperparameters. The results of the hyperparameter search and other hyperparameter setups are provided in Appendix C.2.
We evaluate all methods through the same protocol of [9. 16. 17], which uses the same amounts of generated images from the reference split specialized for each dataset.? Besides, we run all the experiments three times with random seeds and report the averaged best performances for reliable evaluation with the lone exception of ImageNet and StyleGAN2 related experiments. Please refer to Appendix C.2 for other experimental details. The numbers in bold-faced denote the best performance and in underline indicate that the values are in one standard deviation from the best.
4.3 Evaluation Results
Comparison with Other cGANs. We compare ReACGAN with previous state-of-the-art cGANs in Tables and 2. We employ the implementations of GANs in PyTorch-StudioGAN library as it provides improved results on standard benchmark datasets [30, 31]. For fair comparison, we provide results from each original paper (denoted as * in method) as well as those from StudioGAN
We use the validation split as the default reference set, but we use the test split of CIFAR10 and the training split of CUB200 and AFHQ due to the absence or lack of the validation dataset.
7

---

Table Experiments using ImageNet [31] dataset. B.S. means batch size for training. (Left): Com- parisons with previous cGAN approaches. (Right): Learning curves of BigGAN [4], ContraGAN [23]. and ReACGAN (ours) which are trained using the batch size of 256.
ImngeNet
Method
15 7
FID L
Fasrs 1
F
and FID scores during training
ACGAN"
184.41
SNGAN" [3]
36.80
27.62
SAGAN [29]
52.52
18.28
BigGAAN
BiGGAN
BigGAN [20]
38,05
22.77
CorezaGAN BoACGAN
ContraGAN
TAC-GAN [20]
23.75
FID (lower ia bettter)
ReACGAN
IS (higher better)
286
ContraGAAN [23]
31.10
19.69
0.951
0.927
SNGAA
ACGAN [18]
62.99
32.25
26.79
26.35
0,935
0.963
14)
SAGAN [29]
29.85
34.73
0.849
0.914
BigGAN [-]
43.97
16.36
0,964
0.955
70-
ContraGAN [23]
25.25
25.16
0.947
0.855
ReACGAN
13.98
BigGAN [4] DiffAug [9]
36.97
18.57
0.956
0.941
50-
ReACGAN DiffAug
69.74
11.95
0,977
0.975
=3= 2018
BigGAN [4]
99.31
8.51
50
BigGAN" [25]
104.57
9.18
ReACGAN
BigGAN [4]
99.71
8:23
7,89
0.991
0.985
0.989
10
1206
14tk 1001:
Real Data
173.33
Iteration
library [40]. We also conduct experiments with popular augmentation- methods: consistency regularization (CR) [16] and differentiable augmentation (DiffAug) [9].
Compared with other cGANs, ReACGAN performs the best most benchmarks, surpassing the previous methods by 2.5%, 5.1%, 15.8%, and 14.5% in FID on CIFARIO, Tiny-ImageNer CUB200, and ImageNet (256 B.S.), respectively. ReACGAN also harmonizes with augmentation-bace regularizations, bringing incremental improvements on all the metrics. For the ImageNet experiments using batch size of 256, ReACGAN reaches higher IS and lower FID with fewer iterations than other models in comparison. Finally, we demonstrate that ReACGAN can learn with larger batch size on ImageNet. While some recent methods [20, 38, 27] have been built on ACGAN improve the generation performance of ACGAN, large-scale image generation experiments with the batch size of 2048 have never been reported. Table shows that our ReACGAN reaches FID score of 8.23 on ImageNet, being comparable with the value of 7.89 from BigGAN implementation in PyTorch-Studine StudioGAAN library. ReACGAN, however, provides better synthesis results than other implementations of BigGAN [4, 25]. Note that our result on ImageNet is obtained in only two runs while the training setup and architecture of BigGAN have been extensively searched and finely tuned.
Comparison with Other Conditioning Losses. We investigate how the generation qualities vary with different conditioning losses while keeping the other configurations fixed. We compare D2D-CE loss with cross cross-entropy loss of ACGAN (AC) [18], loss used in the projection discriminator (PD) [19], multi-hinge loss (MH) [26], and conditional contrastive loss (2C) [23]. The results are shown in Table 3. AC- and MH-based models present decent results on CIFAR 10, but undergo early-training collapse on Tiny- ImageNet and CUB200 datasets. Replacing them with PD, 2C, and D2D-CE losses produce satisfactory performances across all datasets, where PD loss makes the best Fs (recall) on CUB200 dataset while giving third 12.125 (precision) value. The noticeable point is that the proposed D2D-CE shows consistent results across all datasets, showing the lowest FID and the highest Fs and Fo.125 values in most cases. This means ReACGAN can generate high-fidelity images and is relatively free from the precision and recall trade-off [43] than the others.
Consistent Performance of ReACGAN on Adversarial Loss Selection. We validate the consistent performance of ReACGAN on four adversarial losses: non-saturation loss [1]. least square loss [10], Wasserstein loss with gradient penalty regularization (W-GP) [13], and hinge loss [46] on CIFARIO and Tiny-ImageNet datasets in Table 4. The experimental results show that BigGAN + D2D-CE (ReACGAN) consistently outperforms the projection discriminator (PD) and conditional contrastive loss (2C) counterparts over three adversarial losses [1, 13, 46]. However, for the experiments using the least square loss [10], ReACGAN exhibits inferior generation performances to the projection discriminator. We speculate that minimizing the least square distance between an adversarial logit and the target scalar (1 or 0) might affect the norms of feature maps and spoil the classifier training performed by D2D-CE loss.
8

---

Table Experiments on the effectiveness of D2D-CE loss compared with other conditioning losses.
CIFARIO [30]
Tiny ImageNet [32]
CUB200 [33]
Conditioning Method
FID
Fauus 1
Fs 1
IS 1
FID
Fauus 1
Fa
1S
FID
4 Fazs f
F4 f
BigGAN w/o Condition [4]
(Abbreviated Big)
9.46
12.21
0.987
0.982
7.38
76.15
0.804
0.576
5.16
35.17
0.852
0.936
Big AC [18]
9.84
8.45
0.993
0.992
6.00
96.04
0.656
0.368
6.19
60.73
0.726
0.891
Big PD [19]
9.81
8.08
0.993
0.992
12.78
32.03
0.948
0.868
18.30
0.924
0.967
Big MH [26]
10.05
7.94
0.994
0.990
4.37
140:74
0.282
0.156
5.18
245.69
0.625
0.832
Big 2C [C33
8.22
0.993
0.991
13.46
0.974
0.881
0.942
Big D2D-CE (ReACGAN)
9.89
7.88
0.994
0.992
14,06
27.10
0.970
0.894
4.91
15.40
0.970
0.954
Table 4: Experiments to identify the consistent performance of D2D-CE on adversarial loss selection.
Adversarial Loss
Conditioning Method
CIFARIO [30]
Tiny ImageNet
IS
FID 4
Fi
Better?
FID
Fa
Fg f Better?
PD [19]
9.75
8.29
0,993
0.991
8.27
58.85
0.816
0.713
Non-saturation [1]
2C [23)
9.30
10:47
0.990
0.959
6.57
84.27
0.745
0.556
D2D-CE
9.79
8.27
0.993
0.991
11.76
39.32
0.942
0.852
PD [19]
9.94
8.26
0.993
0.992
12.74
37.14
0.920
0,900
v
Least square [10]
[23]
12.18
0.986
0,941
9.58
0.706
D2D-CE
9.70
9.56
0.991
0.987
9.50
57.67
0.848
0.692
PD [19]
5.71
64.75
0.792
0.652
6.67
84.16
0.696
0.498
W-GP [13]
2C [23]
5.93
55.99
0.842
0.709
6.89
74.45
0.812
0.536
D2D-CE
7,30
35.94
0,942
0.847
8.92
$2.74
0.856
0.689
PD [19]
9.81
8.08
0.993
0,992
12.78
32.03
0.948
0.868
Hinge [46]
2C [23]
9.70
8.22
0.993
0.991
13.46
28.55
0.974
0.881
D2D CE
9.89
7.88
0.994
0.992
14.06
27.10
0.970
0.894
Table 5: FID [34] values of conditional StyleGAN2 (eStyleGAN2) [7] and StyleGAN2 [7] D2D-CE
on CIFAR10 [30] and AFHQ [41] datasets. ADA means the adaptive discriminator augmentation [8]. FID is exceptionally computed using the training split for calculating the reference moments since FID value StyleGAN2 is often calculated using the moments of the training dataset.
Conditioning method
CIFAR1O [30]
AFHQ [41]
cStyleGAN2 [7]
3.88
StyleGAN2 [7] D2D-CE
3.34
cStyleGAN2 [7] ADA [S]
2.43
4.99
StyleGAN2 [7] ADA [8] D2D-CE
2.38
4.95
StyleGAN2 [7] DirtAug [9] D2D-CE Tuning
2.26
Effect of D2D-CE for Different GAN Architectures. We study the effect of D2D-CE loss with different GAN architectures. In Table 5, we validate that D2D-CE is effective for StyleGAN2 [7] backbone and also fits well with the adaptive discriminator augmentation (ADA) [8]. StyleGAN2 with D2D-CE loss produces 13.9% better generation result than the conditional version of StyleGAN2 (cStyleGAN2) on CIFAR 10. Moreover, StyleGAN2 with D2D-CE can be reinforced with ADA or DiffAug when train StyleGAN2 + D2D-CE under the limited data situation. Among GANs, StyleGAN2 DiffAug D2D-CE + Tuning achieves the best performance on CIFARIO, even outperforming some diffusion-based methods [47, 48]. Additional results with other architectures, i.e., a deep convolutional network [2] and a resnet style backbone [13], are provided in Appendix E.
4.4 Ablation Study
We study how each component of ReACGAN affects ACGAN training. By adding or ablating each part of ReACGAN, as shown in Table 6, we identify four major observations. (1) Feature and weight normalization greatly stabilize ACGAN training and improve generation performances on Tiny-ImageNet and CUB200 datasets. (2) D2D-CE enhances the generation performance by considering data-to-data relationships and by performing easy sample suppression (3th and 4th rows). (3) the suppression technique does not work well on the feature normalized cross-entropy loss (5th row). (4) While ACGAN shows high Inception score on ImageNet experiment, it shows relatively

---

AFHQ -112
IMAGENRT 128
CUB200 1282
LLNY-MMAEENT
CIFARIO 322
Figure 5: Curated images generated by the proposed ReACGAN. More qualitative results on ReAC-
GAN, BigGAN [4], ContraGAN [23], and ACGAN [18] are attached Appendix J.
Table 6: Ablation study on feature map normalization, data-to-data consideration, and easy positive and negative sample suppression.
MnnyImmaeNNe Net [32]
CUB200 [33]
ImageNet [49]
Ablation
IS f
FID
Fo.12
Fs
IS f
FID
Fo,12
IS 1
FID |
Fauzs 1
Fg
ACGAN [18]
6.00
96.04
0.656
0.368
6.09
60.73
0.726
0.891
62.99
26.35
0.935
0.963
+ Normulization
13,46
30.33
0.955
0.889
4.78
25.54
0.883
0.952
18.16
36.40
0.879
0.787
+ Data-to-data (Eq. (5))
12.96
28.71
0.967
0.863
5.08
25.12
0.894
0.946
Suppression (Eq. (6))
14.06
27.10
0.970
0.894
15.40
0.954
63.16
0.974
0.974
Data-to-data
12.96
30.79
0.960
0.857
5.39
30.36
0.863
0.947
poor FID, 125, and F3 values compared with the model trained with normalization, data-to-data consideration, and the suppression technique. This result is consistent with the qualitative results on ImageNet, where the images from ACGAN are easily classifiable, but the images from ReACGAN are high quality and diverse. We attribute the improvement to the discriminator that successfully leverages informative data-to-data and data-to-class relationships with casy sample suppression.
Conclusion
In this paper, have analyzed why training ACGAN becomes unstable the number of classes in the dataset increases. By deriving the analytic form of gradient in the classifier and numerically checking the gradient values, exploding problem caused by the unboundedness of input feature vectors and poor classification abil- ity of the classifier in the early training stage. To alleviate the instability and reinforce ACGAN, we have proposed the Data-to-Data Cross-Entropy loss (D2D-CE) and the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN). The experimental results verify the superiority of GANs on five benchmark
have discovered that the unstable training comes from a gradient
ReACGAN compared with the existing classifier- and projection-barte datasets. Moreover, exhaustive analyses on ReACGAN prove that ReACGAN is robust to hyperpa- rameter selection and harmonizes with various architectures and differentiable augmentations.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the IITP grants NN.2001--001196: AI Graduate School Program POSTECH, N0.2021-0-00537: Visual Common Sense, No.2021-0-02068: AI Innovation Hub) funded by Ministry of Science and ICT, Korea.
References
[1] Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances Neural Information Processing Systems (NewrIPS), pages 2672-2680, 2014.
[2] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation I. Learning with Deep Convolutional Generative Adversarial Networks. arXiv preprint arXiv 1511.06434, 2016.
[3] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Gen- erative Adversarial Networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[5] Tero Karras, Samuli Laine, and Timo Aila, A style-bused generator architecture for generative adver- sarial networks. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 1401-4410, 2019.
[6] Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy P. Lillicrap. LOGAN: Latent Optimisation for Generative Adversarial Networks. arXiv preprint arXiv 1912,00953, 2019.
[7] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lchtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 81 10-8119, 2020.
[8] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training Generative Adversarial Networks with Limited Data. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[9] Shengyu Zhao, Zhijian Liu, Ji Lin, Yan Zhu, and Song Han. Diffcrentiable augmentation for data- efficient gan training. arXiv preprint arXiv 10738, 2020.
[10] Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhixiang Wang, and Stephen Paul Smolley. Least Squares Generative Adversarial Networks. In Proceedings of the International Conference Computer Vision (ICCV), pages 2813 2221222,,220 2017.
[11] Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv preprint arXiv 1701.07875. 2017.
[12] Martín Arjovsky and Léon Bottou. Towards Principled Methods for Training Generative Adversarial Networks. In Proceedings of the International Conference Learning Representations (ICLR), 2017.
[13] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved Training of Wasserstein GANs. In Advances in Neural Information Processing Systems (NeurIPS), pages 5767-5777, 2017.
[14] Naveen Kodali, James Hays, Jacob D. Aberntthy, and Zsolt Kira. On Convergence and Stability of GANs. arXiv preprint arXiv 1705.07215, 2018
[15] Zhiming Zhou, Jiadong Liang. Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. Lipschitz generative adversarial nets. In Proceedings of the International Conference on Machine Learning (ICML), pages 7584-7593, 2019.
[16] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lec. Consistency Regularivation for Generative Adversarial Networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[17] Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved Consistency Regularization for GANs. arXiv preprint arXiv 2002.04724, 2020.
11

---

[18] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional Image Synthesis with Auxiliary Classifier GANs. In Proceedings of the International Conference on Machine Learning (ICML), pages 2642-2651, 2017.
[19] Takeru Miyato and Masanori Koyama. cGANs with Projection Discriminator. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[20] Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin Auxilary Classifiers GAN. In Advances in Neural Information Processing Systems (Neur/PS), 2019.
[21] Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening and Coloring Batch Transform for GANs. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[22] Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few- Shot Unsupervised Image-to-Image Translation. In Proceedings of the International Conference on Computer Vision (ICCV). 2019.
[23] Minguk Kang and Jaesik Park. ContraGAN: Contrastive Learning for Conditional Image Generation. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[24] Woohyeon Shim and Minsu Cho. CircleGAN: Generative Adversarial Learning across Spherical Circles. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[25] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian, Omni-GAN: On the Secrets of cGANs and Beyond. arXiv preprint arXiv:2011. 13074, 2020.
[26] Ilya Kavalerov, Wojeiech Czaja, and Rama Chellappa. A multi-class hinge loss for conditional gans. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021.
[27] Liang Hou, Qi Cao, Huawei Shen, and Xueqi Cheng. eGANs with Auxiliary Diseriminative Classifier. arXiv preprint arXiv:2107. 10000 2021.
[28] Han, Ligong and Min, Martin Renqiang and Stathopoulos, Anastasis and Tian, Yu and Gao, Ruijiang and Kadav, Asim and Metaxas, Dimitris N. Dual Projection Generative Adversarial Networks for Conditional Image Generation. In Proceedings of the International Conference Computer Vision (ICCV), 2021.
[29] Han Zhang, Ian Goodfellow, Dimitris Metaxas. and Augustus Odena. Self-Attention Generative Adversarial Networks. Proceedings of the International Conference on Machine Learning (ICML). pages 7354-7363, 2019.
[30] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. PhD thesis, University of Toronto, 2012.
[31] Jin Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei ImageNet: A large-scale hierarchical image database. In Proceedings the IEFF International Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[32] Johnson et al. Tiny ImageNet Visual Recognition Challenge. https://ttin- hher-/eemeeeeeeeeeeuuuaaaa com.
[33] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSI Birds 200. Technical report, California Institute of Technology. 2010.
[34] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances Neural Information Processing Systems (NeurIPS), pages 6626-6637, 2017.
[35] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization. In Advances in Neural Information Processing Systems (NeurIPS), pages 271-279 2016.
[36] Akash Srivastava, Lazar Valkov, Chris Russell, Michael Gutmann, and Charles Sutton. VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Leaminng In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[37] Mehdi Mirza and Simon Osindero, Conditional Generative Adversarial Nets. arXiv preprint arXiv 1414.1784, 2014.
12

---

[38] Zhiming Zhou, Han Cai, Shu Rong. Yuxuan Song, Kan Ren, Weinan Zhang, Jun Wang, and Yong Yu. Activation Maximization Generative Adversarial Nets, In Proceedings of the International Conference on Learning Reprexentations (ICLR), 2018.
[39] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A Simple Framework for Contrastive Learning Visual Representations. arXiv preprint arXiv 2002.05709, 2020.
[40] Minguk Kang and Jaesik Park. Pytorch-studiogan. httpp:///githbb cot.://TECC CVLab/
SSSdiiiGAA,, 2020.
[41] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. StarGAN v2: Diverse Image Synthesis for Multiple Domains. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Jung-Won
[42] Tim Salimans, Goodfellow, Wajeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Chen. Im- proved Techniques for Training GANs. In Advances in Neural Information Processing Systems (NeurIPS), pages 2242- 2016.
[43] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[44] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. Learned Representation For Artistic Style. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
[45] Harm de Vries, Florian Strub, Jeremie Mary. Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. Advances Neural Information Processing Systems (NeurlPS), pages 6594-6604, 2017.
[46] Jac Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv preprint arXiv 1705.02894, 2017.
[47] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the International Conference on Machine Learning (ICML). 2021.
[48] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Score Matching Model Unbounded Data Score. arXiv preprint 2rXiv:2106..0522, 2021.
2021.
[49] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos 7afeirion. Areface: Additive angular margin loss deep face recognition. In Proceedings of the IEFE International Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
[Yes] The main contributions of our paper can be summarized into two points: (1) analyzing why ACGAN training becomes unstable the number of classes in the training dataset increases, (2) developing a new cGAN model: the Rebooted Auxil- iary Classifier Generative Adversarial Networks (ReACGAN), Both claims are well reflected in the abstract and introduction.
(b) Did you describe the limitations of your work?
[Yes] The proposed ReACGAN generates high-quality images using informative data- to-data and data-to-class relationships in the dataset. However, it requires additional hyperparameter tuning for a temperature and margins values. Also, we analyze the weakness of ReACGAN in Appendix F.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Generative Adversarial Networks can be used to synthesize indistinguishable images from the real. This might cause fake news, sexual harnesses, and hacking machine vision applications. Although the potential negative social impacts are not discussed in our main paper, we deal with this issue in Appendix G.
(d) Have you read the ethics review and ensured that your paper conforms to them? [Yes] We carefully read the ethics review guidelines and comply with them.
13

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results?
[Yes] We state all assumptions that arise from the theoretical results, i.e., Property 1-4 Sec. 3.3 in the main paper.
(b) Did you include complete proofs of all theoretical results?
[Yes] We provide the complete proofs in Appendix D.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)?
[Yes] We release model weights and the software package used in paper at www.o//SOOTTCCLLO//////.. Torch-StudioGiAN.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We state details for training models Sec. 4.2 in the main paper, which include the evaluation procedure, hyperparameter selection, and controlled environment for a fair comparison. In addition, we add how we pre-process training datasets in the Appendix C.I.
(c) Did you report
bars (e.g., with respect the random seed after running experi- ments multiple times)?
all the experiments three times with random seeds. We provide standard deviations of Tables 1, 3, 4, Appendix I.
[Yes] For reliable evaluation,
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)?
[Yes] We include the total amount compute and the type of resources Appendix H.
4. If you using existing assets (e-g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite papers and GitHub repository that we used for our experiments in Sec. 4.2.
(b) Did you mention the license of the assets?
[No] We utilize five datasets validate our method: CIFARIO, Tiny-ImageNer CUB200, ImageNet, and AFHQ. These are widely used public datasets, and we cite them in the paper. However, we could not find any license information for CIFAR 10, and CUB200. The ImageNet and AFHQ datasets can be used for non- commercial purposes only. addition, we mention the license of Py TorchhSttuiiodee in the main paper, which is used for our experiments.
(c) Did you include any new assets either in the supplemental material or as a URL? Yes] We provide the URL of our source code the abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating?
[Yes] The CIFAR10. TingeImag,Net, CUB200, ImageNet, and AFHQ datasets are properly curated, and they are popular datasets for research purpose. To our best knowledge, there no inappropriate contents in the datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] The benchmark datasets we used are widely adopted by others and do contain any harmful or troublesome contents.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] Our research has no relevance to the crowdsourcing human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable?
[N/A] Our research has relevance to the crowdsourcing human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] Our research has no relevance to the crowdsourcing or human subjects.
14


Reviews:
[{"review_id": "caBRfbggH5f", "reviewer": "Reviewer_t8Ys", "summary": "This paper tackles limitation of ACGAN.\n1. ACGAN struggles when # classes is large.\n2. ACGAN generates easily classifiable samples (=lacks diversity).\n\nTh authors argue that\n1. Gradient exploding in the classifier leads to collapse. Normalizing input vectors to the classifier resolve it.\n2. data-to-data cross-entropy exploits relational information (?).\n\nPerformance is measured in IS, FID, PR on cifar10, cub, tiny-imagenet, and imagnet.\n(Misc.) ReACGAN benefits from consistency regularization and diffaug.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality.\n+ (+) Showing that the feature norm and the gradient (dL/dw) norm has connection to the performance is interesting.\n+ (+)Combining the standard cross-entropy for classification and contrastive loss is novel (eq. 5).\n+ (+)Difference between ReACGAN and ContraGAN is clearly described.\n- (-) Normalizing the representations for computing inner product has already been standard in representation learning (simclr, mocov2, clip ...). They should be covered as related work.\n\nQuality.\n+ (+) Hyperparameter search is provided in the appendix. FID is mostly robust to the choices.\n- (-) ACGAN's early training collapse is not rigorously defined and not shown either empirically or theoretically.\n- (-) Incorrect argument: As $F(x_i)^Tw_k$ arises, p gets closer to 1, (1-p) gets closer to 0, and the norm of the gradient shrinks.\n> L108 The norm of the gradient begins to explode as the inner-product value of $F(x_i)^Tw_k$ arises\n- (-) The paper does not cover multi-task discriminator which has is a strong standard in recent methods [FUNIT, StarGANv2, Which training methods do actually converge]. Describing that the multi-task discriminator does not have similar phenomena will make the argument on ACGAN's instability stronger.\n- (-) Reasons for using DiffAug and not using [ADA] are not described.\n\n\nClarity.\n+ (+) The proposed method is clearly described enough to be implemented.\n- (-) Thresholds m_p and m_n should be termed margins instead of thresholds, following [Learning shallow convolutional feature descriptors with triplet losses].\n- (-) The proposed method is referred to as ReACGAN and Big+D2D-CE interchangeably. Using a single term is recommended to avoid confusion.\n\nSignificance.\n+ (+) The proposed method is effective on ACGAN and ProjectionGAN variants.\n+ (+) The proposed method synergies with consistency regularization.\n\nMain reasons for my rating toward rejection come from logical weaknesses. Please refer to the <Quality> section.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They mentioned necessity of the hyperparameter search as the limitation.\nIn my opinion, it is not important to describe potential negative societal impact because the paper tries to improve general quality of GANs.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ZjzZFGMKIFI", "reviewer": "Reviewer_BEb7", "summary": "This paper presents a method to tackle the training instability issue caused by auxiliary classification loss in ACGAN. To this end, the authors propose to modify the function to estimate the class conditional probability by replacing the denominator of the softmax function with the sum of the inner product between the reference image and its negative examples. In addition, the authors introduced margin to limit the norm of the gradients. Experiments are performed on multiple well-known benchmarks such as Imagenet and compared with several previous methods.  The metric used to compare the methods are widely used ones such as FID and Inception score. Experimental results show that the proposed is superior compared to the existing methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper motivates with identifying the gradient explosion problem especially in the early training phase in the ACGAN due to the auxiliary loss and convincingly presents the idea to tackle the problem.\n\nThe paper is well written and easy to follow. All the notations, facts, and figures are clear to understand.\n\nThe experiments are extensive and the theoretical justifications seem correct. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper addresses the issue with the existing GANs frameworks in order to improve the quality of image generations. Hence, I don't foresee any new negative impact that GAN  this paper can cause.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Z6moAcCzi5Y", "reviewer": "Reviewer_ojXE", "summary": "In this paper, the authors are focused on a particular class of Generative Adversarial Networks (GANs) , the conditional GANs (cGANs) which incorporating additional information into GANs.\nMore specifically, they focus on auxiliary classifier GAN with softmax cross-entropy loss (ACGANs) \nACGANs suffer from challenging training as the number of classes in the dataset increases and also from generating easily classifiable samples without considering diversity.\nThe authors identify the source of the first problem, namely they show that gradient exploding in the classifier can cause an undesirable early training collapse, and resolve it via projecting input vectors onto a unit hypersphere. \nThey also propose Data-to-Data Cross-Entropy loss (D2D-CE) to exploit fruitful relational information in the class-labeled dataset. \nTo the end, they propose the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN).\nThey also perform experimental study which show that ReACGANs achieve state-of-the-art generation performances on several datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors analyse why ACGAN training becomes unstable as the number of classes increases. \nThen they propose how to solve this problem : \n1) by analytically calculating gradients of the softmax cross-entropy loss used in ACGAN. By examining the exact values of analytic gradients, they show that the unboundedness of input feature vectors and poor classification performance in the early training phase can cause an undesirable gradient exploding problem;\n2) having alleviated the instability, they propose the Rebooted Auxiliary Classifier Generative Adversarial Networks (ReACGAN) using the Data-to-data Cross-Entropy loss (D2D-CE). ReACGAN projects image embeddings and proxies onto a unit hypersphere and computes similarities for data-to-data and data-to-class consideration. Additionally, they introduce two threshold values for inter-class and intra-class separability. In this way, ReACGAN overcomes the training instability and can exploit supervisory signals by explicitly considering data-to-class and data-to-data relationships, and also by implicitly looking at class-to-class relationships in the same mini-batch.\n\nThe authors do indeed identify and resolve a major problem of ACGAN but their contribution to the problem is incremental in the sense that when compared with the existing state of the art methods, their performance is similar.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\n1) The hyperparameters (\\tau, m_p, m_n) seem to play an important role both on the training and on the performance of the algorithm. \nThe authors say that identify their values through grid search. It would be useful to see the perrformance of algorithm for different values, as well -- ie to see how robust it is wrt different values.\nAlso, m_p=1-m_n seems convenient but not justifiable;\n\n2) In Table two there are some underlined values; could please explain what the underlying means? Also, in the same Table we see that none of the algorithms attains IS close to that of the real data at the Tiny-Imagenet dataset; do the authors have any explanation about that?\n \n3) Given that the paper contains graphs, I would advice the authors to employ colourblind-friendly colours so that they can be accessible from colour-blind fellows;", "review_text": "", "extra_scores": {"ethical_concerns": "-", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "Ex99-hKlE4A", "reviewer": "Reviewer_HH86", "summary": "This paper proposes the remedy for issues present in ACGAN formulation which causes mode collapse.  The major contributions are:\n- Comparison to relevant state-of-the-art baselines has been done.\n- The paper is lucidly written and the authors have done extensive experiments.  \n- Code for the proposed method has been provided by the authors. Great effort has been made by authors to ensure reproducibility.\n\nHowever paper claims to have rebooted ACGANs but the D2D-CE discriminator used, seems dissimilar to the ACGAN and seems more similar to contrastive loss. In my opinion, I feel there is a clear mismatch between claims and results. \n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Clarity\n\nThis paper claims to Reboot ACGANs but in the paper the best performance is achieved by D2D-CE loss which is very similar to the contrastive loss used in contraGAN. The auxiliary classifier in D2D-CE performs inferior to the ACGAN classifier as shown in Appendix Table A9, which clearly shows that the discriminator with D2D-CE loss is definitely not an auxiliary classifier (as it has poor classification performance compared to ACGAN). I do appreciate the finding that normalization of embedding helps in improvement of ACGAN, but the final model is clearly dissimilar to ACGAN. Hence in my opinion the claim of rebooting ACGAN is a bit misleading.\n\n\n### Originality: \n\nI find that a lot of ideas in this paper have already been explored in the ContraGAN [K1] paper\n\n1. Idea of Normalization: The idea of normalization of embeddings such that they lie on a sphere is also used by ContraGAN.\n2. Idea of Positive Negative Repulsion: The idea of introducing a positive negative based repulsion seems similar to the contrastive loss in ContraGAN.\n3. Eq.5 in the paper is very similar to Eq. 7 in ContraGAN paper which is the base equation both papers build on.\n4. The authors have introduced D2D-CE which seems to be inspired from hard mining based contrastive losses and afaik is novel in GANs. \nThe works in appendix references [25,26,27] already have introduced the hard mining losses for classification, hence I recommend to use these losses as well and show a comparison, to make the submission more stronger.\n\n\n### Quality \n\nThere have been a lot of experiments done for verifying the effectiveness of the proposed method. I find that gains in performance for the CIFAR-10 and Tiny Imagenet overlap in the standard deviation intervals, hence the claim of state-of-the-art cannot be fully verified on these datasets. The technique seems to perform better in case of the CUB dataset.\n\n### Significance \n\nI find that a lot of the ideas stated in this paper have already been explored in ContraGAN paper. Also the proposed loss is in my opinion, is more similar to contrastive loss rather than ACGAN loss. The description of the current work as Rebooting ACGAN seems misleading as the D2D-CE method is a contrastive method based on hard-mining of samples.\n\n\n[K1] Kang M, Park J. Contragan: Contrastive learning for conditional image generation. arXiv preprint arXiv:2006.12681. 2020 Jun 23.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I feel that limitations haven’t been discussed appropriately in the manuscript. A line of description on limitations is definitely not enough. More discussion on additional training time due to the new loss. Also some things like how much time was required for choosing hyperparameters can also be discussed in limitations.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors identify that the auxiliary classifier GAN (ACGAN) fails to scale to more classes and is biased towards easily classifiable examples because ACGAN can suffer from gradient explosion when training the classifier, causing the collapse of training. They then propose to normalize the feature vectors and add a data-to-data loss function to improve the performance. The resulting model is named Rebooted ACGAN (ReACGAN), which can be easily combined with other techniques like consistency regularization for better performance. Experimental results reveal some competitive performance of ReACGAN.

The reviewers generally appreciate the clarity of the paper and the convincing argument from the claimed issue of ACGAN to the improved model. One issue that could be addressed easily is to rigorously define and describe what the authors mean by early training collapse. The major issue that was not easily addressed is on how the authors can establish originality with respect to

(1) the widely-used normalization technique in other GANs and representative learning works. The authors argue that their in-depth investigation, instead of the more ad-hoc use of other works, is an important contribution. While the contribution is indeed of some value, the authors only position this investigation within the narrow angle of the ACGAN model. It is not clear whether the contribution can also be generally used to explain the normalization technique used in other GANs. This narrow angle limits the impact of the contribution.

(2) the ContraGAN work which has a contrastive loss that shares some similarity to the proposed data-to-data loss function. While some reviewers believe that the difference between ReACGAN and ContraGAN is clearly described, and the authors argue that ReACGAN and ContraGAN losses are designed with different principles (Line 177), other reviewers point out that more efforts are needed to highlight the differences between the two loss functions, such as empirical studies on how "false negative samples" or other loss variants affect performance (like the preliminary results provided in the rebuttal). While the authors' arguments on how ReACGAN originated from ACGAN (rather than ContraGAN) is convincing, they do need more efforts to clarify whether the ReACGAN is significantly different to ContraGAN in their final form to claim for originality. Some efforts are also needed to smoothly incorporate the rebuttal paragraphs into the revised paper.

One issue that some reviewers and authors disagree on is whether ReACGAN should be "called" ReACGAN, given that the classifier nature in ACGAN was deliberately weakened for ReACGAN and the proposed loss makes the whole model look a lot more like ContraGAN. The naming issue is arguably more subjective and is not taken as the key factor behind the final decision making. Somehwo the authors are encourage to keep studying a related issue on analyzing how the classifier performance affects conditional generation performance.

The authors addressed some of the concerns with preliminary experiments, such as the comparison to multi-task discriminators to ensure whether ReACGAN is competitive against those, or the comparison to other hard-mining-based loss functions. The experiment results clarified some concerns, but need more efforts to be blended to the main paper.



Decision: Accept (Poster)