A Novel Confidence Guided Training Method for Conditional GANs with Auxiliary Classifier

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 05:19:08

Abstract: Conditional Generative Adversarial Network (cGAN) is an important type of GAN which is often equipped with an auxiliary classifier. However, existing cGANs usually have the issue of mode collapse which can incur unstable performance in practice. In this paper, we propose a novel stable training method for cGANs with well preserving the generation fidelity and diversity. Our key ideas are designing efficient adversarial training strategies for the auxiliary classifier and mitigating the overconfidence issue caused by the cross-entropy loss. We propose a classifier-based cGAN called Confidence Guided Generative Adversarial Networks (CG-GAN) by introducing the adversarial training to a  $K$-way classifier. In particular, we show in theory that the obtained $K$-way classifier can encourage the generator to learn the real joint distribution. To further enhance the performance and stability, we propose to establish a high-entropy prior label distribution for the generated data and incorporate a reverse KL divergence term into the minimax loss of CG-GAN. Through a comprehensive set of experiments on the popular benchmark datasets, including the large-scale dataset ImageNet, we demonstrate the advantages of our proposed method over several state-of-the-art cGANs.

Main paper: 
 introduction: 
 Generative Adversarial Network (GAN) [7] is a popular generative model for high-fidelity image generation, which has been extensively studied in recent years [1, 18, 27, 33] . Though other generative models, such as the diffusion models [4, 10] , have recently also attracted a lot of attentions due to their effectiveness in generating high-quality images, GANs still enjoy several significant advantages in practical applications, such as their lower computational complexities for training and inference [14, 15] . The key idea of GAN is to simultaneously train a generator and a discriminator by using the adversarial game: the generator takes random noise to generate the fake data so as to fool the discriminator, meanwhile, the discriminator tries to distinguish between the real and fake data.During this adversarial training process, the generator becomes stronger and can generate new data (e.g., images) with high quality. The original GANs do not utilize the label information (e.g., the labels of the train data). Mirza and Osindero [26] proposed the conditional GANs (cGANs) that utilize the label information to generate some specific class of data. For example, one can condition the cGANs with the labels for animals to generate the images of dogs and cats separately. cGANs have been used for various applications, such as text-to-image generation [15, 31] , image style transformation [39] , and speech enhancement [25] .In general, most existing cGANs can be classified into two main categories, the classifier-based cGANs [2, 11, 30] and the projectionbased cGANs (which incorporate the conditional information into the discriminator) [8, 28] . For a classifier-based cGAN, it often takes advantage of a classifier to utilize the class information. For example, the classifier can penalize the mismatched data-label pairs during the training process [11, 13] . As a representative classifierbased cGAN, the "Auxiliary classifier GAN (AC-GAN)" proposed by Odena et al. [30] contains an auxiliary classifier to learn a conditional label distribution for guiding the generator to generate class-specific images. Although these proposed cGANs can achieve promising generation quality, recent researches have shown that they often suffer from two problems in practice: (1) the performance of the generator drops at the early training stage (i.e., early-training collapse), particularly when working with datasets that have a large number of classes [11, 13, 28, 37] ; (2) the generator tends to generate data with low diversity [11, 28] .To improve the performance of cGANs, a number of elegant methods have been proposed, which mainly focus on modifying the network structure or the loss function of the classifier. For example, to remedy the early-training collapse issue, ReACGAN [13] normalizes both the input feature vectors and the weight vectors in the classifier; Hou et al. [11] proposed ADC-GAN that is based on an auxiliary discriminative classifier for achieving better training stability and generation diversity (we provide a detailed introduction on more existing approaches in Section 5). Despite of the improvements achieved by these methods, their practical performances are still not quite satisfying in some scenarios. For example, in Figures 1a and 1b , we illustrate the Inception Score (IS) [32] and Fréchet Inception Distance (FID) [9] curves of AC-GAN and several improved models on Tiny-ImageNet [23] , where IS and FID are commonly used metrics for evaluating the performance in terms of generation fidelity and diversity. We can see that the performances of some cGAN methods decrease after a certain number of iterations. In Figure 1c , we show the classification accuracies on generated images of several cGAN methods on the large-scale dataset Imagenet [3] ; some of their conditional generation performances are relatively low. Figure 1 : (a) and (b) are the IS (higher is better) and FID (lower is better) curves on Tiny-ImageNet [23] . The figures contains the curves of PD-GAN [28] , AC-GAN [30] , AM-GAN [38] , Multi-hinge (MH) GAN [20] , ReACGAN [13] , and ADC-GAN [11] ; the "rCG-GAN" is our proposed cGAN method in Section 4. (c) Conditional image generation results on ImageNet [3] . The "acc" (higher is better) means the ImageNet classification top-1 accuracy on the generated images, which reveals the conditional generation performance. The numerical results of AC-GAN, PD-GAN and ReACGAN are reported from [13] .Our contributions. From the above discussion and the experimental results shown in Figure 1 , we can see that the major challenge for designing a promising cGAN is to achieve stable and high-quality generation as well as high conditional generation performance. We design an efficient adversarial training strategy for the auxiliary classifier and propose a novel conditional GAN method based on the aforementioned AC-GAN, which is called "Confidence Guided Generative Adversarial Networks (CG-GAN)".We demonstrate theoretically that the 𝐾-way classifier of CG-GAN can encourage the generator to approach the real joint distribution. By analyzing the gradient of the classification loss in CG-GAN, we elucidate the mechanism through which our CG-GAN improves training stability. Furthermore, we investigate the potential challenges that CG-GAN may encounter in practical applications. To mitigate these issues, we propose the establishment high-entropy prior label distribution for the generated data. Subsequently, we incorporate the corresponding inverse Kullback-Leibler (KL) divergence term into the minimax loss of CG-GAN, leading to enhanced stability and performance. We refer to the CG-GANs with reverse KL divergence terms as "rCG-GAN". The potential limitations of the two most closely related works are analyzed, (i.e., ReACGAN [13] and ADC-GAN [11] ), to highlight the advantage of rCG-GAN.To validate the effectiveness of our method, a set of experiments on several popular banchmark datasets are conducted, including CIFAR10, CIFAR100 [21] , Tiny-ImageNet [23] , Baby/Papa/Grandpa-ImageNet [14] and ImageNet [3] . Compared with the recently proposed classifier-based and projection-based GANs, our rCG-GAN can achieve better performance in terms of the IS and FID scores over those benchmark datasets. Notably, for the large-scale dataset ImageNet, rCG-GAN can achieve significant improvement over the previous state-of-the-art methods: rCG-GAN yields the IS that is at least two times of their IS scores, and the FID that is only half of theirs. Moreover, rCG-GAN has better top-1 and top-5 classification accuracies on generated images, which indicate the improvement on conditional generation performance.The rest of this paper is organized as follows. In Section 2, we overview the definitions for GAN and the related cGANs with auxiliary classifier. We conclude that adversarial training is often missing applied to the auxiliary classifier in these cGAN models. Then, we explore the relationship between early-training collapse and over-confidence. In Section 3, we show our method for training classifier-based cGANs. In Section 4, we illustrate our experimental results and the comparisons with several state-of-the-art cGANs. Finally, we discuss the related work and conclude in Section 5 and Section 6, respectively. 
preliminaries: 
 Generative Adversarial Networks. Let 𝑋 be the data space. The original GAN [7] consists of two neural networks: the generator 𝐺 that maps a given random noise 𝑧 to a generated data point 𝑥 ∈ 𝑋 , and the discriminator 𝐷 that distinguishes between real data and generated data by mapping each data point 𝑥 ∈ 𝑋 to a value in [0, 1]. Denote by 𝑃 𝑋 and 𝑄 𝑋 the real data distribution and the generated data distribution, respectively. The adversarial training is to optimize the following losses:EQUATIONAC-GAN. The Auxiliary classifier GAN (AC-GAN) [30] is one of the most representative Classifier-based cGANs, which uses an auxiliary classifier to improve the performance of the ordinary GAN. The objective of AC-GAN also consists of two parts as the GAN losses Equation (1) and Equation (2) , where the difference is that they both contain the penalty items for the classification loss.Given the training dataset with 𝐾 classes (𝐾 ∈ Z + ), a 𝐾-way classifier "𝑙" maps each input 𝑥 ∈ 𝑋 to the label space R 𝐾 . Let 𝑌 = {1, 2, • • • , 𝐾 }. For each 𝑦 ∈ 𝑌 , let 𝑙 𝑦 (𝑥) denote the 𝑦-th element of 𝑙 (𝑥) corresponding to the label 𝑦. A widely used loss for classifier is the softmax cross-entropy loss with one-hot encoding:EQUATIONwhere Pr(𝑦|𝑥) is the conditional probability of the ground-truth label 𝑦. For ease of presentation, we in particular call Pr(𝑦|𝑥) the "confidence" of 𝑦; namely, the confidence indicates the probability that 𝑥 belongs to class 𝑦. In Section 3, we will introduce our cGAN model based on Pr(𝑦|𝑥), and that is why we name it as "confidence guided cGAN". Let 𝑃 𝑋𝑌 (resp., 𝑄 𝑋𝑌 ) denote the joint distribution of the real (resp., generated) data and labels in 𝑋 × 𝑌 . The objective functions of AC-GAN are defined as follows:EQUATIONminEQUATIONwhere 𝐿 𝐷 and 𝐿 𝐺 are defined in Equation ( 1) and Equation ( 2), and 𝜆 > 0 is the given coefficient.Adversarial training missing on the auxiliary classifier. MH-GAN [20] improves upon AC-GAN by substituting the crossentropy loss with a multi-class extension of the widely used hinge loss. ReACGAN [13] suggests normalizing feature embeddings onto a unit hypersphere to address the early-training collapse problem, and expanded the cross-entropy loss of the classifier to the data-to-data cross-entropy loss. We conclude that most classifierbased cGANs, such as AC-GAN, MH-GAN and ReACGAN, lack efficient adversarial training on the auxiliary classifier. This deficiency makes it challenging for generators to learn the real joint distribution of the training data, thereby diminishing the diversity of the generated samples.Early-training collapse and over-confidence. As mentioned before, conditional GANs with auxiliary classifier are prone to early-training collapse. An important observation of Kang et al. [13] is that the unboundedness of input feature norm can cause undesirable gradient explosion problem for the classifier of AC-GAN, which usually leads to early-training collapse. Also, as discussed in several papers [16, 35] , large feature norm is a main reason for over-confidence (i.e., peaky prediction distribution), because the confidence is usually proportional to its feature norm and it turns to encourage the classifier to always output increasingly large feature norm so as to encourage high confidence.Based on these insights, in the Section 3, we focus on designing novel classification loss functions which incorporate effective adversarial training strategies and addressing the issues of early-training collapse and over-confidence. 
our proposed training method: 
 In this section, we propose a novel stable adversarial training method for the classifier-based cGANs. First, we introduce our basic model "CG-GAN", and explain that why CG-GAN can encourage the generator to learn the real joint distribution optimally in Section 3.1. Then, we study the gradient of the classification loss of CG-GAN in Section 3.2, which is the key to improve the training stability. In Section 3.3, we discuss the challenges for training the basic CG-GAN and propose an improved version "rCG-GAN" , which can be implemented more efficiently in practice. Finally, we discuss distinctions between rCG-GAN and closely related works, analyzing potential limitations of ReACGAN and ADC-GAN [11] to highlight advantages of rCG-GAN.Our high-level idea. To introduce the adversarial training to a classifier, a natural idea is to encourage the discriminator to return higher confidence for the output from real data and lower confidence for the output from generated data. This intuition is somewhat similar to the adversarial strategy for training a standard GAN [7] . However, we focus on the classifier and need to develop some significant new ideas for the loss function with the analysis from the stability perspective. The key of implementing our idea is to design an appropriate classification loss for achieving an effective balance between the adversarial training and the confidence.In the stage of optimizing the discriminator, we consider minimizing the softmax cross-entropy loss (i.e, 𝜎 sce (𝑥, 𝑦) in Equation ( 3)) on real data and maximizing 𝜎 sce (𝑥, 𝑦) on generated data. Specifically, we intend to minimize the following classification loss in the training procedure for the discriminator:EQUATIONIt is worth noting that directly optimizing the loss Equation ( 6) may result in a technical issue in practice: the value of 𝜎 sce (𝑥, 𝑦) on generated data can be quite large, making the training process challenging to converge and undermining the performance of the classifier. To avoid this issue, we incorporate a hinge loss and introduce an upper bound "𝑚 > 0" to our model. Since "𝜎 sce (𝑥, 𝑦)" depends on the confidence Pr(𝑦|𝑥) (see Equation ( 3)) and the introduced parameter 𝑚 can restrict the confidence, we name this model as "Confidence Guided Generative Adversarial Network (CG-GAN)". Formally, the objective functions for the classifier of CG-GAN in the discriminator and generator are defined as follows:EQUATIONwhere the notation[𝑎] + = 𝑎 if 𝑎 ≥ 0, otherwise, [𝑎] + = 0.The second term of Equation ( 7) serves as a crucial aspect during optimization, indicating that when the value of 𝜎 sce (𝑥, 𝑦) on generated data exceeds the value of 𝑚, it will not further increase. This thresholding mechanism helps to mitigate the issue of degradation of classifier performance, a potential challenge that can be encountered in adversarial training settings. Coupled with the GAN losses, our CG-GAN has the following objectives:EQUATIONminEQUATIONWe propose the following Proposition 3.1, which states the conditions under which the training objective of the classifier of CG-GAN achieves the global optimum. Proposition 3.1 reveals that the classifier encourages the generator to learn the real joint distribution. Due to the space limit, we place the proof to our supplement. As discussed in Section 2, a major reason for early-training collapse and over-confidence is from the unboundedness of feature norm and its induced gradient explosion problem. We explain why our CG-GAN can improve the training stability from this perspective. We focus on the classifier of CG-GAN in the discriminator (i.e., the objective function Equation ( 7)). Note that the value 𝑙 𝑦 (𝑥) in 𝜎 sce (𝑥, 𝑦) (Equation ( 3)) can be written as a dot-product 𝑓 (𝑥) ⊤ 𝑤 𝑦 , where 𝑓 (𝑥) is the feature embedding vector and 𝑤 𝑦 is the weight vector of the classifier associated with class 𝑦 in the last fully connected layer. Suppose we sample 𝑛 real data samples from the given training data, and 𝑛 generated data samples from the generator.For each 1 ≤ 𝑖 ≤ 𝑛, let 𝑥 𝑟 𝑖 and 𝑥 𝑔 𝑖 denote the 𝑖-th real and 𝑖-th generate samples, respectively. In practice, we use the empirical cross-entropy loss to represent 𝐶 𝑐𝑔 𝑑 in Equation ( 7): First, we defineEQUATIONwhere Pr(𝑦|𝑥 Intuitive analysis from Lemma 3.2: Lemma 3.2 reveals that when the confidence of the classifier for the generated data surpasses exp(-𝑚), the gradient of 𝐿 𝑤 (𝑖) should be equal to 𝐺 𝑟 (𝑖, 𝑘) -𝐺 𝑔 (𝑖, 𝑘). Moreover, once the confidence exceeds exp(-𝑚) (i.e., 𝑚 -𝜎 sce (𝑥, 𝑦) > 0), the objective function described in Equation (7) encourages the classifier to exhibit low confidence on the generated data. Since Proposition 3.1 reveals that the classifier encourages the generator to learn the real joint distribution, 𝑓 (𝑥 𝑔 𝑖 ) is guided to be close to 𝑓 (𝑥 𝑟 𝑖 ) and suppressing the confidence of the classifier on the generated data implicitly affects the confidence on the real data. Note that the gradient norm of the cross-entropy loss in AC-GAN is always equal to ||𝑓 (𝑥 𝑟 𝑖 )|| 1 𝑦=𝑘 -Pr(𝑦|𝑥 𝑟 𝑖 ) , and the norm ||𝑓 (𝑥 𝑟 𝑖 )|| of AC-GAN is encouraged to increase to obtain high confidence. Consequently, the norm of the gradient ||𝐺 𝑟 (𝑖, 𝑘) -𝐺 𝑔 (𝑖, 𝑘)|| in our CG-GAN should be less than the norm of gradient in AC-GAN, which can alleviate the gradient explosion problem and thus improve the training stability. We place the proof of Lemma 3.2 in supplement.Potential over-confidence issue. To implement our proposed CG-GAN as described in Section 3.1, it is crucial to consider the optimal confidence function of Equation (3) when the training objective for the CG-GAN classifier reaches its global optimum. Specifically, we denote the optimal softmax cross-entropy function of Equation ( 3) as 𝜎 * sce (𝑥, 𝑦) and the corresponding optimal confidence function as Pr * (𝑦|𝑥). We propose the following corollary: Corollary 3.3. When the training objective for the classifier of CG-GAN achieves the global optimum, the optimal confidence function Pr * (𝑦|𝑥) may be any value between exp(-𝑚) and 1.We place the proof to our supplement. Corollary 3.3 reveals the optimal confidence function of CG-GAN may be close to 1, which correspondingly means that the feature norm of the sample will become large. As discussed in Section 2, large feature norm can lead to unstable training. As shown in Figure 2 , our experimental findings observed that the CG-GAN model exhibited instability after long training. This observation highlights the necessity of introducing a regularization term to effectively address the issue of over-confidence in CG-GAN.Classifier criteria. Before addressing the issue of overconfidence in CG-GAN, it is also important to consider the classification criteria of the auxiliary classifier. First, we clarify two necessary conditions for a qualified classifier. Suppose 𝑃 (𝑥) = {𝑝 1 (𝑥), 𝑝 2 (𝑥), • • • , 𝑝 𝐾 (𝑥)} is the prediction distribution by the classifier with the softmax activation function. For any input data 𝑥 with label 𝑦, the classifier should satisfy:EQUATION(ii) 𝑝 𝑦 (𝑥) ≥ 𝑝 𝑘 (𝑥) for any 𝑘 ≠ 𝑦,where𝑝 𝑦 (𝑥) = exp 𝑙 𝑦 (𝑥 ) 𝐾 𝑘=1 exp 𝑙 𝑘 (𝑥 )and𝑝 𝑘 (𝑥) = exp 𝑙 𝑘 (𝑥 ) 𝐾 𝑘=1 exp 𝑙 𝑘 (𝑥 ). Based on Corollary 3.3, we have 𝑝 𝑦 (𝑥) = Pr * (𝑦|𝑥) ≥ exp(-𝑚). We just simply let 𝑚 ≤ log(𝐾), and then we have 𝑝 𝑦 (𝑥) = Pr * (𝑦|𝑥) ≥ exp(-𝑚) ≥ 1 𝐾 . As a consequence, the classifier of CG-GAN satisfies the condition (12) . However it cannot guarantee the condition (13) . In particular, the negative softmax cross-entropy in Equation (7) minimizes 𝑝 𝑦 (𝑥) = Pr(𝑦|𝑥) = exp 𝑙 𝑦 (𝑥 ) 𝐾 𝑘=1 exp 𝑙 𝑘 (𝑥 ) (i.e, minimizes 𝑙 𝑦 (𝑥) while maximize 𝑙 𝑘≠𝑦 (𝑥)) on the generated data, which may lead to 𝑝 𝑦 (𝑥) < 𝑝 𝑦≠𝑘 (𝑥) (i.e., 𝑙 𝑦 (𝑥) < 𝑙 𝑘≠𝑦 (𝑥)).Our proposed solutions. To better control the behavior of the classifier on the generated data with respect to the two conditions ( 12) and ( 13), we define the following prior label distribution:P = 1-exp(-𝑚) 𝐾 -1 ,. .The 𝑦-th item ,. . .,EQUATION)From the assumption 𝑚 ≤ log(𝐾), we have exp(-𝑚)≥ 1 𝐾 ≥ 1-exp(-𝑚) 𝐾 -1; then we can minimize the KL-divergence between the prior label distribution P and the prediction distribution 𝑃 (𝑥) on the generated data to encourage the classifier to satisfy the two conditions ( 12) and (13) . Moreover, the KL-divergence term can serve as a regularization mechanism to help mitigate the aforementioned over-confidence issue. Hence, we can add the expected value of KL(𝑃 (𝑥)|| P) or KL( P ||𝑃 (𝑥)) to Equation (7) . To better balance the losses of the discriminator and generator, we also add the expected value of -KL(𝑃 (𝑥)|| P) or -KL( P ||𝑃 (𝑥)) to Equation (8) . We propose rCG-GAN: the CG-GAN with the where 𝜆 1 , 𝜆 2 > 0 are two given coefficients. Similarly, we can also have fCG-GAN: the CG-GAN with the forward KL divergence, where we just simply replace the term KL(𝑃 (𝑥)|| P) with KL( P ||𝑃 (𝑥)) in Equation ( 15) and Equation ( 16). In Section 4, we mainly focus on rCG-GAN since fCG-GAN often achieves similar experimental results.Remark 1. Note that the regularization term, specifically the KL term, is explicitly applied to the generated data. Nonetheless, it is crucial to acknowledge that suppressing confidence in the generated data can indirectly influence confidence in the real data. This occurs as the classifier encourages the generator to learn the real joint distribution, as indicated in Proposition 3.1. Such a feedback mechanism may lead to a decreased feature norm in both the real and generated data, consequently improving the stability of the CG-GAN. We provide the experimental verification of this phenomenon in the supplement.Differences between CG-GAN and EBGAN. EBGAN [36] introduces an energy-based formulation for unconditional GANs, whereas our CG-GAN is specifically designed to address the issues commonly encountered in training instability and overconfidence in Classifier-based cGANs. Unlike EBGAN, CG-GAN incorporates an auxiliary classifier and employs label information and the softmax cross-entropy function. By analyzing the gradient of the classification loss in CG-GAN, we can explicate why our model improves training stability. Moreover, EBGAN has a fundamentally different network structure for the discriminator, which is based on an auto-encoder.Analysis on ReACGAN. ReACGAN [13] employs the normalization of both feature embeddings and weight vectors to address the collapse issue. However, this method may have limited improvement on the condition generation performance compared to AC-GAN. Furthermore, the absence of adversarial training on the classifier of ReACGAN may make it challenging to ensure the generator to effectively learn the real data distribution, consequently diminishing the diversity of the generated samples.Analysis on ADC-GAN. To better explain the possible problems of the auxiliary discriminative classifier GAN (ADC-GAN) [11] in the practical optimization, we discuss the mathematical expression of the objective functions of ADC-GAN below. It introduces the "discriminative classifier" which can play both the roles of the classifier and discriminator. In particular, for a training dataset with 𝐾 classes, the discriminative classifier 𝑙 maps each 𝑥 ∈ 𝑋 to R 2𝐾 , where 𝐾 dimensions serve for the 𝐾 labels of real data, and the other 𝐾 dimensions serve for the 𝐾 fake labels of generated data. To distinguish the real and fake labels, we add the superscripts "𝑟 " (for real data) and "𝑔" (for generated data) to each ground-truth label 𝑦 ∈ 𝑌 . So "𝑙 𝑦 𝑟 (𝑥)" denotes the 𝑦 𝑟 -th element of 𝑙 (𝑥) corresponding to the real ground-truth label 𝑦 𝑟 , and similarly, "𝑙 𝑦 𝑔 (𝑥)" denotes the 𝑦 𝑔 -th element of 𝑙 (𝑥) corresponding to the fake ground-truth label 𝑦 𝑔 . Given a data 𝑥 ∈ 𝑋 , Pr(𝑦 𝑟 |𝑥) denotes the confidence that it is a real data and has the label 𝑦 𝑟 ; and Pr(𝑦 𝑔 |𝑥) can be defined similarly. Also, similar with Equation (3) we haveEQUATION)EQUATIONThe discriminative classifier tries to minimize the following Equation (19) and Equation (20) :𝐶 𝑑 = -E 𝑥,𝑦∼𝑃 𝑋 𝑌 [log Pr(𝑦 𝑟 |𝑥)] -E 𝑥,𝑦∼𝑄 𝑋 𝑌 [log Pr(𝑦 𝑔 |𝑥)]; (19) 𝐶 𝑔 = -E 𝑥,𝑦∼𝑄 𝑋 𝑌 [log Pr(𝑦 𝑟 |𝑥)] + E 𝑥,𝑦∼𝑄 𝑋 𝑌 [log Pr(𝑦 𝑔 |𝑥)]. (20)Then we can simplify Equation (20) :EQUATIONIn the optimization discriminator phase, the loss function, i.e., Equation (19) , guides the classifier to act as a discriminator by distinguishing differences between real and generated data. However, the dependency between generated data and the real label is absent since the same class of real and generated data must be segregated into two distinct classes, namely the "real" and "fake" labels. This dependency is only provided during generator optimization via the loss function represented by Equation (21) . Nevertheless, due to the absence of a cross-entropy loss function when optimizing the generator in Equation ( 21), the conditional generation ability of ADC-GAN may not be sufficiently trained. 
experiments: 
 We compare our proposed rCG-GAN (and fCG-GAN) with several recently proposed cGANs with BigGAN [1] backbone, including AC-GAN [30] , PD-GAN [28] , ReACGAN [13] and ADC-GAN [11] .We use the open-source library StudioGAN repository to conduct our experiments 1 .Datasets and evaluation metrics. We consider five public datasets: CIFAR10 [21] (60k images of 10 classes), CIFAR100 [21] (60k images of 100 classes), Tiny-ImageNet [23] (120k images of 200 classes), Baby/Papa/Grandpa-ImageNet [14] (each has 100 classes), and ImageNet [3] (1,281k and 50k images for training and validation with 1k classes). Baby/Papa/Grandpa-ImageNet are created by StudioGAN [14] for small-scale ImageNet experiments.We consider six evaluation metrics: "Inception Score (IS)" [32] , "Fréchet Inception Distance (FID)" [9] , "Density" and "Coverage" [29] , and the "improved Precision" and "improved Recall" [22] . IS and FID are widely used metrics for evaluating the performance of generative models. Some studies [1, 37] show that IS tends to measure the generation fidelity and FID tends to capture the diversity of the generated data. Density and Coverage are a pair of metrics designed to disentangle the fidelity and diversity measurement from FID. The improved Precision and Recall [22] are proposed to overcome the shortcomings of the original Precision and Recall for the generated distribution against the real data distribution.Some experimental details. We use the validation dataset as the default reference distribution for the computing of evaluation metrics. For CIFAR10 and CIFAR100, we use the test dataset due to the absence of the validation dataset. We adopt the default configurations of the ReACGAN paper [13] in StudioGAN and follow [5, 11] to use the hinge loss [24] for the implementation of the GAN losses 𝐿 𝐷 and 𝐿 𝐺 . The number of training iterations is set to 100, 000 for CIFAR10/CIFAR100 and 200, 000 for the other five datasets; the batch size is set to 64 if not specified. The parameters 𝜆 1 and 𝜆 2 for our rCG-GAN and fCG-GAN are set to 1.0 (we also investigated the performance of rCG-GAN with different values of 𝜆 1 and 𝜆 2 , and we place the results to our supplement due to the space limit).Note that the performances of rCG-GAN and fCG-GAN are similar in practice, so we mainly focus on rCG-GAN in our experiments except for the ablation study. Recall that our proposed CG-GAN depends on the parameter 𝑚; in fact the value exp(-𝑚) is the desired confidence for Pr * (𝑦|𝑥). So for convenience, the value exp(-𝑚) is called by "desired confidence" in our experiments. According to our discussion in Section 3.3, we should set exp(-𝑚) ≥ 1/𝐾 for each testing dataset. Particularly, we also study the experiment with varying the value exp(-𝑚) at the end of this section.Ablation study. We conduct the experiments to show that the necessity of coupling the CG-GAN with the KL(𝑃 (𝑥)|| P) or KL( P ||𝑃 (𝑥)) (as discussed in Section 3.3), for improving the performance of CG-GAN and avoiding early-training collapse. As shown in Table 1 , both fCG-GAN and rCG-GAN outperform CG-GAN in terms of six metrics on CIFAR10 and CIFAR100, and we observe that rCG-GAN performs slightly better than fCG-GAN with respect to FID. In the second part of our ablation study, we further examine the effectiveness of CG-GAN, fCG-GAN, and rCG-GAN in terms of training stability. Let "AC-GAN + rKL" and "AC-GAN + fKL" denote the methods of AC-GAN with KL(𝑃 (𝑥)|| P) and KL( P ||𝑃 (𝑥)), respectively. As shown in Figure 2 , our CG-GAN, which is equipped with the adversarially trained auxiliary classifier, can achieve better stability comparing with the baselines "AC-GAN + rKL", "AC-GAN + fKL", ACGAN [30] and AMGAN [38] . Moreover, our proposed rCG-GAN and fCG-GAN can successfully avoid early-training collapse and exhibit superior stability compared to the basic CG-GAN.Comparison with existing cGANs. We illustrate our experimental results on CIFAR-10, CIFAR-100, Tiny ImageNet and Baby /Papa/Grandpa-ImageNet in Table 2 . From the results we can see that our rCG-GAN achieves the best of 5 scores (except for the improved Recall) on the datasets: the scores of IS, Density and improved Precision are used for measuring the generation fidelity; FID and Coverage are used for measuring the diversity of the generated images. So the results shown in Table 2 suggest that our proposed method yields better generation fidelity and diversity which are important metrics for evaluating the performance of generative model. Some qualitative results of our rCG-GAN and baseline approaches are shown in supplement due to the space limit.Image generation on ImageNet. To further evaluate the performance of our method for large-scale dataset, we also consider the experiment on ImageNet, which is a widely used dataset in the computer vision community. We conduct the experiment on ImageNet with 128 × 128 image resolution in the batch size of 256, and the results are shown in Table 3 . Our rCG-GAN outperforms other cGANs by a large margin in terms of IS and FID; moreover, we also report the classification accuracies and our advantage is also significant. The results indicate that rCG-GAN not only achieves high-quality image generation but also demonstrates promising performance in terms of conditional image generation. We also place several generated images in our supplement.Conditional generation performance on datasets with different levels of classification difficulty. To have a more comprehensive comparison of the conditioning performance between rCG-GAN and other cGANs, we evaluate the Top-1 and Top-5 classification accuracies on generated images using the pretrained Inception-V3 network [34] on three subsets of ImageNet: Baby/Papa/Grandpa-ImageNet. These subsets were created by StudioGAN [14] based on the classification difficulty of the images. Baby-ImageNet represents the easiest subset to classify, while Grandpa-ImageNet represents the most difficult subset. From the results shown in Table 4 , we can see that our rCG-GAN can achieve the best conditional generation performance in terms of classification accuracies as well as FID, across the datasets with different classification difficulty levels.Impact of the desired confidence. In practice, we set exp(-𝑚) to be slighly larger than 1 𝐾 , where 𝐾 represents the number of classes in the dataset. By this setting, we can satisfy both condition ( 12) and ( 13) to guide the conditional generation in the training procedure. Additionally, a moderately small value of exp(-𝑚) leads to a high entropy distribution in (14) , that is helpful to mitigate the over-confidence issue. We conduct the experiments with varying the desired confidence exp(-𝑚). As shown in Figure 3 , our experiments on CIFAR100 indicate that the change of exp(-𝑚) yields a mild trade-off between Improved Precision/IS and Improved Recall: the Improved Recall decreases as the Improved Precision and IS score increases, when we vary exp(-𝑚) from 0.011 to 0.05. For the  
related work: 
 PD-GAN [28] is a representative projection-based cGAN that incorporates class information into the discriminator by learning an embedding for each class. As a representative classifier-based cGAN, AC-GAN [30] uses an auxiliary classifier appended to the discriminator, and added the cross entropy loss to the standard GAN loss. Zhou et al. [38] introduced the AM-GAN, which employs a (𝑘 + 1)-way classifier along with extra "fake" labels for supervised learning. TAC-GAN [6] introduces a twin classifier to address the biased learning objective of AC-GAN. ContraGAN [12] applies the conditional contrastive loss and the cross-entropy loss to capture the data-to-data relationship and the data-to-label relationship. ECGAN [2] presents a comprehensive outlook on cGANs classification accuracies (%) on the generated images using the pre-trained Inception-V3 network, respectively. * : the results reported by the each original paper ; † : the results reported by [13] ; ‡ : the results reported by [14] . https://github.com/POSTECH-CVLab/PyTorch-StudioGAN 
 
 Appendix:  


Decision: After the rebuttal, three reviewers (EhLb, xnx3, and g5sU) reached a consensus to accept the paper due to its innovation, extensive experimental results, and in-depth theoretical analysis. Reviewer jYoz assigned a Borderline Reject because of doubts about the paper's innovation, but still agrees to the paper being accepted. The rebuttal effectively addressed the major concerns raised by the reviewers. After reading the paper, its rebuttal, and the discussion, the AC finds the paper well-written, sound, and well-justified by the experiments. Thus, the AC recommends accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel conditional generative adversarial network training method called Confidence Guided Generative Adversarial Networks (CG-GAN), aiming to address the issue of mode collapse in existing cGANs training processes and enhance the diversity and quality of generated samples. The authors devise an efficient adversarial training strategy tailored to the auxiliary classifier and propose a novel high-entropy prior label distribution, enhancing the stability and performance of CG-GAN through a reverse KL divergence term. Experimental results on multiple public datasets demonstrate significant advantages of CG-GAN over state-of-the-art cGANs in terms of generation quality and diversity evaluation metrics such as Inception Score (IS) and Fréchet Inception Distance (FID), particularly showcasing remarkable performance on large-scale dataset ImageNet. Additionally, the paper explores the impact of expected confidence on model performance and demonstrates its robustness across datasets with varying levels of classification difficulty through experiments. Overall, this work provides valuable insights and methodologies for improving the training stability and generation performance of cGANs.
strength_weakness: strength_weakness: 1. Innovation: The paper introduces a novel cGAN training method, namely CG-GAN, which addresses mode collapse and overconfidence issues by introducing adversarial training with an auxiliary classifier. This represents a novel research direction in the field of GANs.
2. Theoretical Analysis: In addition to proposing a new method, the author provides in-depth theoretical analysis, including demonstrating that the auxiliary classifier in CG-GAN encourages the generator to learn the conditional real joint distribution, thereby enhancing the academic depth of the study.
3. Experimental Validation: The paper conducts experimental validation on multiple widely recognized benchmark datasets, including the large-scale ImageNet dataset. The experimental results indicate that the proposed method exhibits advantages in both generation quality and diversity.
limitations: limitations: 1. In Figure 1(a), the trend of rCG-GAN continues to rise within 200 iterations. Will it keep increasing with further iterations? Why doesn't the graph extend to the convergence point of rCG-GAN? Does it keep increasing indefinitely?
2. Technical Transparency: The paper may not sufficiently elaborate on the technical details of the new method, particularly regarding the specific implementation of adversarial training strategies and the design choices for the classifier. Such information is crucial for replicating experiments and further research.
3. Unclear Structure: The paper's structure lacks clarity, and the main theme is not sufficiently defined. Related work and the innovative contributions of the paper are intertwined, making it difficult to emphasize the key points.
4. Lack of Visual Representation: As a paper proposing a novel model, there is a lack of visual aids such as model diagrams or flowcharts for overall clarification.
5. Generalization Verification: The experiments mainly focus on specific cGAN architectures, lacking sufficient discussion on the generalization capabilities of the new method across different types of cGAN architectures or various application scenarios.
6. Computational Efficiency: Although the new method shows excellent performance in improving generation quality, the paper lacks discussion on computational efficiency and resource consumption. This is crucial for evaluating the practicality and scalability of the method.
7. The paper should provide generated instances within the main text. Existing evaluation metrics for generation do not adequately assess the real visual effects. Generated instances aid in better evaluating the performance of the method.
paper_summary: paper_summary: The paper addresses the issue of mode collapse and unstable performance in conditional Generative Adversarial Networks (cGANs) equipped with an auxiliary classifier. The proposed method introduces Confidence Guided Generative Adversarial Networks (CG-GAN), which leverage adversarial training strategies for the classifier to preserve generation fidelity and diversity. By integrating a high-entropy prior label distribution and a reverse KL divergence term into the minimax loss, the method enhances the performance and stability of cGANs. Extensive experiments on benchmark datasets like ImageNet demonstrate the advantages of this approach over existing state-of-the-art cGAN models.
strength_weakness: strength_weakness: **S1 Comprehensive Testing:** The effectiveness of CG-GAN is demonstrated through rigorous testing across multiple datasets, including large-scale benchmarks like ImageNet. The use of widely recognized metrics (Inception Score and Fréchet Inception Distance) further validates the improvements in both diversity and fidelity of generated images.

**S2 Theoretical Support:** The paper provides a theoretical basis for the proposed methods, explaining how the changes to the loss functions and the integration of adversarial training can lead to better learning of the real joint distribution of data and labels. This theoretical insight is crucial for understanding the impact of the proposed modifications.

**S3 Practical Improvements:** The proposed CG-GAN shows significant improvements over existing methods in terms of image quality and training stability. This is particularly important for applications requiring reliable and diverse image generation, such as in media creation or data augmentation for training other machine learning models.
limitations: limitations: **W1 Novelty Concerns:**

Auxiliary Classifier in GANs: The concept of integrating an auxiliary classifier within GAN architectures is not new. The Auxiliary Classifier GAN (AC-GAN) introduced by Odena et al. in 2016 ("Conditional image synthesis with auxiliary classifier GANs.") already utilized this concept to improve the quality of generated images by conditioning on class labels. The proposed CG-GAN method enhances this idea by introducing adversarial training for the classifier and a novel loss function involving reverse KL divergence, but the foundational concept remains similar to previous works. This might raise questions about the degree of innovation in the approach.
Improvements on Existing Concepts: While the paper introduces enhancements like high-entropy label distribution and specific loss adjustments for stability and fidelity, these are fundamentally incremental improvements over existing classifier-based GAN frameworks. For example, the use of adversarial training in the context of GANs has been extensively explored in literature, and while the specific application to the classifier component is somewhat novel, it builds directly on established methods.
paper_summary: paper_summary: The authors propose a novel and stable training method to improve the performance and stability of classifier-based conditional Generative Adversarial Networks (cGANs). The key idea is to design an efficient adversarial training strategy for the auxiliary classifier. They also propose to establish a high-entropy prior label distribution for the generated data and inposie a reverse KL divergence regularization term. The introduced method helps to mitigate the over-confidence issue caused by the classifier.
strength_weakness: strength_weakness: - The high-level idea of this paper is insightful. Introducing the adversarial training to the auxiliary classifier is novel and makes sense.
- The paper is well-written and easy to follow.
- The experiments are comprehensive, and the experimental results show that the proposed method can significantly outperform the relevant methods.
limitations: limitations: - My main concern is the proof for proposition 3.1. In Equ. (24), the optimal Q_{XY} is obtained via the classification loss. However, as shown in Equ. (7), it is also necessary to consider L_G to obtain the optimal Q_{XY}. As the authors claimed for their high-level idea, 'This intuition is somewhat similar to the adversarial strategy for training a standard GAN [7],' I am interested in seeing a proof like the one in [7].
- The authors claim that the proposed method can mitigate the over-confidence issue, but the paper lacks experimental results showing the classification confidence scores to demonstrate this.
paper_summary: paper_summary: This paper introduces a novel stable training method for conditional Generative Adversarial Networks (cGANs) to preserve generation fidelity and diversity, addressing the common issue of mode collapse. By incorporating adversarial training into a K-way classifier, this method aims to enhance the generator's ability to learn the real joint distribution. Extensive experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Introduced a novel confidence guided generative adversarial network (CG-GAN) and provides a solid theoretical analysis demonstrating how the proposed method encourages the generator to learn the real joint distribution of data.   
2. Incorporated a prior label distribution and KL divergence term into the loss function, solving the problem of instability of CG-GAN after long-term training.
limitations: limitations: 1. From the experimental results, this method seems to work well, and perhaps the open source code can be provided.   
2. Why can adding KL divergence to the loss function solve the shortcomings of CG-GAN? Please provide a more detailed introduction or proof.  
3. How does this method solve the problem of mode collapse? The author should propose corresponding experiments and theories to elaborate on this.  
4. I would like to know the additional time and computational cost brought by this method.