Class Balancing GAN with a Classifier in the Loop
Authors: Harsh Rangwani
Konda Reddy Mopuri
Venkatesh Babu Radhakrishnan
Venue:  (2021)
Keywords: Long-tailed Learning
GAN
Universal Adversarial Perturbations

Abstract: Generative Adversarial Networks (GANs) have swiftly evolved to imitate increasingly complex image distributions. However, majority of the developments focus on performance of GANs on balanced datasets. We find that the existing GANs and their training regimes which work well on balanced datasets fail to be effective in case of imbalanced (i.e. long-tailed) datasets. In this work we introduce a novel and theoretically motivated Class Balancing regularizer for training GANs. Our regularizer makes use of the knowledge from a pre-trained classifier to ensure balanced learning of all the classes in the dataset. This is achieved via modelling the effective class frequency based on the exponential forgetting observed in neural networks and encouraging the GAN to focus on underrepresented classes. We demonstrate the utility of our contribution in two diverse scenarios: (i) Learning representations for long-tailed distributions, where we achieve better performance than existing approaches, and (ii) Generation of Universal Adversarial Perturbations (UAPs) in the data-free scenario for the large scale datasets, where we bridge the gap between data-driven and data-free approaches for crafting UAPs.

Under review as a conference paper at ICLR 2021
CLASS BALANCING GAN WITH A CLASSIFIER IN THE LOOP
Anonymous authors Paper under double-blind review
ABSTRACT
Generative Adversarial Networks (GANs) have swiftly evolved to imitate increas- ingly complex image distributions. However, majority of the developments focus performance of GANs on balanced datasets. We find that the existing GANs and their training regimes which work well on balanced datasets fail to be effee- tive in case of imbalanced (i.e. long-tailed) datasets. In this work introduce novel and theoretically motivated Class Balancing regularizer for training GANs. Our regularizer makes of the knowledge from pre-trained classifier ensure balanced learning of all the classes in the dataset. This is achieved via modelling the effective class frequency based on the exponential forgetting observed in neu- ral networks and encouraging the GAN to focus on underrepresented classes. We demonstrate the utility of our contribution in two diverse scenarios: (i) Learning representations for long-tailed distributions, where we achieve better performance than existing approaches. and (ii) Generation of Universal Adversarial Perturba- tions (UAPs) in the data-free scenario for the large scale datasets, where we bridge the gap between data-driven and data- free approaches for crafting UAPs.
INTRODUCTION
Image Generation witnessed unprecedented success in recent years following the invention of Gener-
ative Adversarial Networks (GANs) by Goodfellow et al. al.] (2014). GANs have improved significantly
over time with the introduction of better architectures (Gulrajani et al. 2217 Radford et al. 2015
formulation of superior objective functions (lolicoeur-Martinauu 2018 Arjovsky al.] | 2017] and
regularization techniques (Miyato et al., 2018). An important breakthrough for GANs has been the ability to effectively use the information of class conditioning for synthesizing images (Mirza & Osindero 2014, Miyato & Koyama| 2018). Conditional GANs have been shown to scale to large datasets such as ImageNet (Deng et al. 2009) with 1000 classes (Miyato & Koyama 2018).
One of the major issues with unconditional GANs has been their inability to produce balanced dis- tributions over all the classes present in the dataset. This is seen as problem of missing modes in the generated distribution. A version of the missing modes problem, known as the 'covariate shift' problem was studied by Santurkar et al. (2018). One possible reason may be the absence of knowl- edge about the class distribution P(Y GANs on the other hand, do not suffer from this issue since the class label Y is supplied to the GAN during training. However, has been recently found by Ravuri & Vinyals (2019) that despite being able to do well on metrics such as Inception Score (IS) (Salimans et al. (2016)) and Frèchet Incep- tion Distance (FID) (Heusel et al. (2017). the samples generated from the sfate-of-the-arr conditional GANs lack diversity in comparison to the underlying training datasets. Further, we observed that although conditional GANs work well in balanced case, they suffer performance degradation in the imbalanced case.
of the generated samples during training. Conditional
In order address these shortcomings, we propose an orthogonal method (with respect to label conditioning) to induce the information about the class distribution P(Y X) of generated samples in the GAN framework using a pre-trained classifier. We achieve this by tracking the class distribution of samples produced by the GAN using a pre-trained classifier. The regularizer utilizes the class distribution to penalize excessive generation of samples from the majority classes, thus enforcing
Here Y represents labels and X represents data.

---

Under review as a conference paper at ICLR 2021
the GAN to generate samples from minority classes. Our regularizer involves novel method of modelling the forgetting of samples by GANs, based on the exponential forgetting observed in neural networks (Kirkpatrick et al. (2017). We infer the implications of our regularizer by a theoretical bound and empirically verify the same.
We conduct empirical analysis of the proposed class balancing regularizer in two diverse and chal- lenging scenarios:
(i) Training GANs for image generation long-tailed datasets: Generally, even in the long taigetalldd distribution tasks, the test set is balanced despite the imbalance in the training set. This is because it important to develop Machine Learning systems that generalize well across all the support regions of the data distribution. avoiding undesired over-fitting to the majority (or head) classes. Hence, it pertinent to train GANs that can faithfully represent all classes.
(ii) Transferring the knowledge from learnt classifier (P(Y (x))) to a GAN being trained on arbi- trary prior distribution p: This is a specific situation where the samples from target distribution X₁ are unavailable. Instead, discriminative feature knowledge is indirectly available in the form of trained classifier ()))) This a perfect fit for crafting input-agnostic (Universal) adversarial perturbations in the data-free scenario. We show that the proposed regularizer can enable the gen- erated samples to not only extract information about the target data with a trained classifier in the loop, but also represent its support to greater extent.
In summary, our contributions can be listed as follows:
We propose 'class-balancing' regularizer that makes use of the statistics (P(Y) (X) of generated samples to promote uniformity while sampling from an unconditional GAN. The effect of our regularizer is depicted both theoretically (Section 3) and empirically (Section 4
We show that our regularizer enables GANs to learn uniformly across classes even when the training distribution is long-tailed. We observe gains in FID and accuracy of classifier trained on generated samples.
We also show that by combining a pre-trained classifier (i.e. P(Y]X;)) trained on a target dataset Xr. with an arbitrary distribution p(X,). our framework is capable of synthesizing novel samples related to the target dataset. We show that UAPs created on such novel samples generalize to real target data and hence lead to an effective data-free attack. This application is novel to our framework and cannot be realized by conditional GANs.
2 BACKGROUND
2.1 GENERATIVE ADVERSARIAL NETWORKS (GANs)
Generative Adversarial Networks (GANs) are formulated as two player game in which the discrim- inator D tries to classify images into two classes: real and fake. The generator G tries to generate images (transforming noise vector ~ P. ) which fool the discriminator (D) into classifying them as real. The game can be formulated by the following objective:
min mar Elog,, + Ez (lg((- G
(I)
The exact optimization for training D is computationally prohibitive in large networks and the GAN trained by alternative minimization using loss functions. Multiple loss functions have been pro- posed for stabilizing the GAN training. In our work we use the relativistic loss function (Jolicoeur- Martineau 2018) which is formulated as:
LTst = - E (D)))))) ((G(2)))
(2)
La
-D())
(3)
This unconditional GAN formulation does not have any class conditioning and produces different number of samples from different classes (Santurkar et al. 2018). In other words, the distribution is not balanced (uniform) across different classes for the generated data.
2

---

Under review as a conference paper at ICLR 2021
2.2
CONDITIONAL GAN
The conditional GAN (Miraa & Osindero using the following objective:
2014) generates images associated with input label
min max Ban [lo Ee [log(1 D(G(=lv))
(4)
G D
The Auxillary Classifier GAN (ACGAN) (Odeaa et al. 2017) uses an auxiliary classifier for classification along with normal discriminator to enforce high confidence samples from the conditioned class y. Whereas cGAN with projection (Miyato & Koyamal 2018) uses Conditional Batch Norm (De Vries et al. 2017) in the generator and uses a projection step in the discriminator to provide class information to the GAN. We refer to this method as cGAN in the subsequent sections.
Possible issue with Conditional GAN in Long-tailed Setting: The objective in eq. 4 can be as learning a different G(zly) and (xrly) for each of the K classes. this case the tail classes with fewer samples practice there is parameter sharing among different class generators but still class specific parameters are also present form of Conditional BatchNorm. We find that performance of conditional GANs degrade more in comparison to unconditonal GANs in the long-tailed scenario (Section
suffer from poor generalization they have very few samples. However, in
Images
100
Percentage of Samples of class
Ameng > 0.9 confidence samples
Among > confidence samples
Real?
No
60
Fale
40
Generstor g
Pru trained Clacs For
Doemibutioo
Reguiriiz
20
Blezive
Cheot
101
10a
10'
107
Eleccive Chass Disxlbution
Effective Number of Samples No (We for class 0)
(a)
(b)
Figure 1: (a) shows the overview of our method and (b) shows the distribution of generated sam- ples of SNDCGAN CIFAR-10 10 for varying values of No- The percentage of class 0 (randomly choosen) samples determined by an annotator (i.e. high accuracy classifier). When No is large, the network tries to decrease fraction of class O samples whereas when No is small it tries to increase fraction of class samples among the generated samples.
METHOD
In our method we propose to introduce a pretrained classifier (C) to provide feedback to the gen- erator about the label distribution P(Y) over the generated images. The proposed regularizer is added with the generator loss and trained using backpropogation. We first describe the method of modelling Section 3.1 The exact formulation of the regularizer and its theoretical properties are described in Section
3.2 The overview of our method is presented in Figure (Ta).
3.1 CLASS STATISTICS FOR GAN
GAN is dynamic system in which the generator G has to continuously adapt itself in way that is able to fool the discriminator D. During the training, discriminator D updates itself, causing the objective for the generator G also change. This change in objective be seen learning of different tasks for the generator G. this context, we draw motivation from the seminal work catastrophic forgetting in neural networks (Kirkpatrick al. 2017) which shows that neural
3

---

Under review as a conference paper at ICLR 2021
Thannin Tistr button
GAI IMIY: 16.80)
(212 44
<ESM (P 48.141
Ours (MC2 95)
234367
Figure 2: Distribution of classes and corresponding FID scores on long-tailed CIFAR- 10 computed on samples generated by GANs with uniform distribution of labels in case of conditional GANs.
network trained using SGD suffers from exponential forgetting of earlier tasks when trained on a new task. Based this, define effective class frequency N%. of class i at cycle as:
N -1 (1-a) + +ck
(5)
Here c/ is the number of samples of class & produced by the GAN in cycle (t - 1). The class of the sample is determined by the pretrained classifier C. Although D gets updated continuously, the update is slow and requires some iterations to change the form of D. Hence we update the statistics after certain number of iterations which compose a cycle. Here a the exponential forgetting factor which is set to 0.5 in all our experiments. We normalize the class frequency No to obtain discrete effective class distribution:
N
N
(6)
Σ N
3.2 REGULARIZER FORMULATION
The regularizer objective is defined as the maximization of the term(Lreg) below:
p& log(pk)
Σ
max P
N
where P = Ei= C(G(s)) In other words, is the average softmax vector (obtained from the classifier C) over the batch of samples and P. is its ttth component corresponding class k. 2; corresponds to random noise vector sampled from Ps. If the classifier C' recognizes the samples confidently with probability 1, P& can be seen as the approximation to the ratio of the number of samples that belong to class & to the total number of samples in the batch n. The N in the regularizer term is obtained through the update rule in Section 3nd a constant during backpropagation. We want to emphasize that classifier C is not required to be trained on same data the GAN, instead can be trained in ways such as cupi-super learning, few-shot learning, etc. For instance, in section 4.2 we show that a classifier trained in semi-supervised scenario also enables the GAN to produce balanced distribution. Hence our approach doesn't 't specifically need labelled data which is in contrast to conditional GANs which require labels for each image while training.
Proposition: The maximization of the proposed objective in leads to the following bound on pi:
- -((log())) Ex NY
P <
(8)
where K is the number of distinct class labels produced by classifier C. Please refer to the appendix Section A. for proof of the same.
Implications of the proposition: The bound p& is inversely related to the exponent of the frac- tion of effective class frequency / En N for a given class In case of generating a balanced distribution, pi 1/K which leads to the exponential average No. 1/K. Hence given suffi- cient iterations, the pı value will achieve the upper bound which signifies tightness of the same. To demonstrate effect of the regularizer empirically, we construct two extreme case examples based on the nature of the bound:
If N > N,, Vi + k. then the bound on p would approach e K ((log(()) 1)-1 Hence the network is expected to decrease the proportion of class i samples.
If N < N₁. Vi * k. then the bound on Pa will be -1 Hence the network is expected to increase the proportion of class lo samples.
4

---

Under review as a conference paper at ICLR 2021
We verified the two extreme cases above by training a SNDCGAN (Miyato al. 2018 (DCGAN
with spectral normalization) on CIFAR-10 10 and fixing Nţ (unnormalized version of ND across time steps and term it as NR. Then initialize N₂ to very large value and very small value. Results presented in Figure (Tb) show that the GAN increases the proportion of samples of class & in case of low N& and decreases the proportion of samples in case of large N₁. This shows the balancing behaviour of proposed regularizer.
3.3 COMBINING THE REGULARIZER AND GAN OBJECTIVE
The regularizer is then combined with the generator loss in the following way:
(9) 2019) that the first term of the loss leads to mini- mization of (P: P.)) that is divergence between real and generated data distribution. The regular- izer term ensures that the distribution of classes across generated samples is uniform. The combined objective provides insight into the working of framework, as the first term ensures that the generated images fall in the image distribution and the second term ensures that the distribution of classes is uniform. As P. comprises of diverse samples from majority class the first objective term ensures that Po is similarly diverse. The second term in the objective ensures that the discriminative prop- erties of all classes are present uniformly in the generated distribution, which ensures that minority classes get benefit of diversity within the majority classes. This is analogous approaches that transfer knowledge from majority to minority classes for taiget classifier learning (Liu et al. 2019b Wang al. 2017).
Lg = - -P...)))(PP...)) (oog)))))lo((((((((() - -(x))) X
It has been recently shown (Jolicour--artineee
EXPERIMENTS
For evaluating the effectiveness of our balancing regularizer, we conduct two sets of experiments: (i) image generation from long-tailed distributions, and (ii) creating Universal Adversarial Pertur- bations in the data-free scenario. The goal of the first task is to generate high quality images across all classes and that of the second task is to craft UAPs when the attacker has no access (c.g. due to privacy) to the target data.
4.1 IMAGE GENERATION FROM LONG-TAILED DISTRIBUTION
In this experiment we aim to learn a GAN over long-tailed dataset, which are prevalent in the real world setting. An important aspect of this problem is that it requires to transfer the knowledge from majority classes to minority classes. Several works have focused on learning classifiers for long- tailed distributions (Cao et al., 2019 Cui et al. 2019). Yet works focusing on Image Generation using long-tailed dataset are limited. Generative Minority Oversampling (GAMO) (Mullick et al.) 2019) attempts to solve the problem by introducing a three player framework. We do not compare our results with GAMO as it is not trivial to extend GAMO to use schemes like Spectral Normaliza- tion, and ResGAN like architecture (Gulrajani et al.) 2017) which impede fair comparison.
We performed our experiments on two datasets, CIFAR-10 and a subset of LSUN. The LSUN subset consists of 250k training images and 1.5k validation images. The LSUN subset is composed of 5 balanced classes; (antuurkr et al. 2018) identify this subset to be challenging case for GANs to generate uniform distribution of classes. The original CIFAR-10 dataset is composed of 50k training images and 10k validation images. We construct the long-tailed version of the imbal- anced dataset by following the same procedure as Cao et al. 2019)). Here, images are removed from training dataset to convert it to long-tailed distribution while the validation set is kept unchanged. The imbalance ratio (p) determines the ratio of number of samples in most populated class to the least populated one: p max (TL, 3/ /ming {n More details can be found in Appendix A.2
Datasets:
Pre-Trained Classifier: An important component of our framework is the pre-trained classifier, a ResNet32 model trained using Deffered Reweighting (DRW) of loss (Cao et al., 2019) on long- tailed versions of LSUN and CIFAR-10 10 datasets. Accuracy of the pre-trained classifiers and training details are present in Appendix A.3
GAN Architecture: We used the SNDCGAN architecture for experiments on CIFAR-10 with im- ages of size of 32 x 32 and SNResGAN (ResNet architecture with spectral normalization) structure
5

---

Under review as a conference paper at ICLR 2021
Imbalance Ranio
100
10
(()
Acc.())
PID(()
KLDiv())
Acc.(T)
(L)
CIFAR-10
SNDCGAN
36,97 0.220
0.31 0.0
68.60
32.53 1: 0.06
0.14 =: 0.0
80.60
27.03 1 0.12
ACGAN
44.10 1: 0.02
0.33 0.0
43.08
38.33 1. 0.10
0.12
1: 0.0
60.01
24,21 1 0.08
<GAN
0.02
47.92
26.09
0.04
0.0
68.34
18.99 +
Ours
32,93 1. 0.11
0.06 1: 0.0
72.96
30.48
1 0.07
0.01 1: 0.0
82.21
25.68.1
0.07
LSUN
SNReEGAN
0.
0.68 + 0.0
75.27
33.28
0.02
0.29 = 0.0
79.20
28.99 = 0.03
ACGAN
43.76 +: 0.06
0.39 = 0.0
62.35
31.98 + 0.02
0.05 + 0.0
75:47
26.43 + 0.04
<GAN
75.39
0.12
0.01
30.68
0.04
0.00
72.93
27.59
Ours
35,04 + 0.19
0.06 0.0
77.93
28.78 + 0.01
0.01 = 0.0
82.13
28.15 = 0.05
Table I: Results on CIFAR- 10 (lop punel) and 5 class subset of LSUN (bottom panel) datasets with varying imbulance. In the last column FID values in balanced secaarios are present for case reference. FID, KL Div. and Acc. are calculated on 50k sampled images from each GAN.
for experiments LSUN dataset with images size of 64 x 64. For the conditional GAN base- lines we conditioned the generator using Conditional BatchNorm. We compare our method to two widely used conditional GANs: ACGAN and cGAN. The other baseline we use is the unconditional GAN (SNDCGAN & SNResGAN) without our regularizer. All the GANs were trained with spectral normalization in the discriminator for stabilization (Miyato al. 2018
Training Setup: We train GANs with learning rate of 0.002 for both generator and discriminator.
We used Adam optimizer with B1 = 0.5 and 82 0.999 for SNDCGAN and 31 = 0 and |3 = 0.999 for SNResGAN. We used a batch size of 256 and 1 discriminator update per generator update. As a sanity check, we use the FID values and visual inspection of images on the balanced dataset and verify the range of values from (Kurach et al. 2019). We update the statistics N by update equation in Section 3.1 after every 2000 iterations. Further details are present in Appendix A.6
Evaluation We used the following evaluation metrics:
KL Divergence from Uniform Distribution of labels: Labels for the generated samples are obtained by using the pre-trained classifier (trained on balanced data) as proxy to annotator.
Classification Accuracy (CA): We use the {(X, Y)} pairs from the GAN generated samples to train a ResNet32 classifier and validate it on real data. For the unconditional GANs the label Y is obtained from the classifier trained on long-tailed data. Note that this is similar to Classifier Accuracy Score (Ravuri& & Vinyals 2019).
Frèchet Inception Distance (FID): measures the 2- 2-Wasserstein Distance on distributions obtained from Inception Network (Heusel et al. 2017). We use 10k samples from CIFAR-10 vali- dation set and 10k (2k from each class) fixed random images from LSUN dataset for measuring FID.
Discussion of Results: We present our results in the following subsections:
1) Stability: In terms of stability we find that cGAN suffers from early collapse in case of high imbalance (p 100) and stop improving under 10k iterations. Though we don't It claim about insta- bility of cGANs in general, we emphasize that the same GAN which stable in balanced scenario is unstable in case of long-tailed version of the same dataset.
2) Biased Distribution: Contrary to cGAN, we find that the distribution of classes generated by ACGAN, SNDCGAN and SNResGAN becomes imbalanced. The images obtained by sampling uniformly and labelling by annotator, suffers from a high KI. divergence to the uniform distribution. This leads to some classes being almost absent from the distribution of generated samples as shown in Figure 2 In this case, in Table [ we observe FID score just differs with small margin even if there is presence of large imbalance in class distribution. Our GAN produces class samples uniformly as is evident from the low KL Divergence.
3) Comparison with -aate-of-the-Att Methods: In this work we also find that classification accu- racy is weakly correlated with FID score which is in agreement to (Ravuri & Vinyals (019). We achieve better classifier accuracy in all cases, better than cGAN which achieves sfate-of-the-art Clas- sifier Accuracy Score (CAS). Our method shows minimal degradation in FID for each long-tailed case, in comparison to the corresponding balanced case. It is also able achieve the best FID in 3 out of 4 long-tailed cases. Hence we expect that methods such as Consistency Regularization (Zhang et al. 2019), Latent Optimization (Wu et al. 2019b) etc. can be applied in conjunction with our method to further improve the quality of images. But in this work we specifically focused on techniques used to provide class information Y of an image X to the GAN, Several of-te-o-thh-an
GANs use an approach similar to cGAN (Wu et al. 2019b Brock et al. 2018) for conditioning the discriminator and the generator.

---

Under review as a conference paper at ICLR 2021
HD
Imbalance Ratio
100
10
CIFAR-10
SNDCGAN
36.97 0.20
0.31 0.0
32.53 1 0.06
0.14 0.0
Oars
FID
KIDDv(
(Supervised)
32,93 1: 0.11
0.06 1: 0.0
30.48 1: 0.07
0.01 = 0.0
SNResGAN ACGAN
30.05
0.0
69.90 + 0.13
0.40 4 0.0
(Semi Supervised)
33.32 0.03
0.14 = 0.0
30.37 0.14
0.04 = 0.0
cGAN
30.87 = 0.06
0.09 = 0.0
LSUN
Ours
28.17 1: 0.06
0.11 1 0.0
SNResGAN
37.70 + 0.10
0.68 0.0
33.28 + 0.02
0.29 = 0.0
Table 3: Results on long-tailed CIFAR-100
(Supervised)
35.04 0.19
0.06 0.0
28.78 1: 0.01
0.01 0.0
dataset with imbalance ratio 10. FID is com- pated through 50k generated images and KL. Div of class distribution of GAN and uniform distribu- present column.
Ours
0.05
0.15
30.96 0.07
0.06 0.0
(Semi Supervised)
Table 2: Comparison of results in Semi Supervised Setting. The pretrained classifier used in our framework is line-tuned with 0.1% of labelled data. The same classilier trained balanced dataset as annotator for calculating K1. Divergence for all haselines.
We also find that our method trained using SNResGAN performs similarly to experiments in Table on long-tailed CIFAR-100 dataset as well. Our method achieves the best FID of 28.17 among all baselines and also achieves balanced class distribution like cGAN. The results are summarized in Table and detailed experimental details are present in Appendix A.6.I
4.2 SEMI-SUPERVISED
GAN
In this section we show that the presence of classifier in our framework is an advantage for it, as it allows classifiers trained through various sources to be used for providing feedback to GAN. This feedback allows the GAN to generate class balanced distributions in cases when the labels for underlying long-tailed distributions are not known. This reduces the need of labelled data in our framework and shows the effectiveness over conditional GAN. As it has been shown in that perfor- mance of conditional GANs deteriote (Lucic al.) 2019) when used with limited labelled data. We use ResNet-50 pretrained model on ImageNet from BiT (Big Image Transfer) (Kolesnikov et al., 2019] and fine tune using 0.1 of labelled data of balanced training set (i.e. images per class for CIFAR-10 and 50 images per class for LSUN dataset). In all long-tailed cases this amount of data for each class is present in the training set.
We observe that with just using 0.1% labelled data are able to obtain significantly balanced distribution as seen by low KI. Divergence in comparison to unconditonal GAN (in Table ? and also achive better FID score than unsupervised GAN. This application is unique to framework as conditional GANs explicity require labels for whole dataset for training. The experimental details present in Appendix A.6
4.3 DATA-FREE UNIVERSAL ADVERSARIAL PERTURBATION
Adversarial perturbation SZegedy et al. (2013) ) is a structured noise added to a benign data sample with aim of confusing the machine learning model processing it, leading to inaccurate infer- ence. Universal Adversarial Perturbations (UAP) (Moosavi-Deztonr that are input agnostic and can fool the model when added to any data sample. These perturbations demonstrate transferability across different deep CNN models posing a challenge to their deploy- ability. Crafting UAPs require original training data on which the target deep model is trained. However, the dataset access can be limited due to privacy restrictions. Attackers overcome this limitation via (i) formulating fate-free objectives (c.g. Mopuri et al. (2017)). or (ii) using a proxy dataset composed of either arbitrary natural samples (c.g. Zhang et al. (2020)) or generated syn- thetic samples (e.g. Moouri et al. (2018b)). GAN inspired generative modelling (Poursaeed et al. 2018): Mopuri et al. (201 8a b)) of the UAPs for a given CNN classifier has been shown to capture these input agnostic vulnerabilities. However, in the absence of the target training data, these models suffer from lack of knowledge about the training distribution. Further, synthetic samples generated using existing methods (e.g. Mopuri et al. (2018))) lack diversity and use an activation maximiza- tion approach which is computationally expensive since optimization has to be performed for each batch of samples separately. To tackle this issue, we introduce an activation maximization term in GAN objective combine discriminative class knowledge (P(Y)X.)) learnt by the classifier
et al. (2017) are such noises
7

---

Under review as a conference paper at ICLR 2021
UAP Algo.
GAN with Regilor ice!
Cip
Tiger S Shark
Gold Fish
Stone Wall
Diverse Images from for different Imagenel Cleeses
Gencreted images aro a proxy for imagenet dota for UAP Algo.
Comioss Datacel (Priot)
Netural Image
Cless
Natural Image
Sea Lion
Figure 3: Overview of our UAP crafting approach using arbitrary data and classifier in the loop- First, with the help of the proposed regularizer, the GAN can generate samples enriched with fea- tures from the ImageNet pre-trained classifier. The UAP algorithm can subsequently craft better perturbations due to the available discriminative information about the target (ImageNet) data.
(C) trained target data (X,) with arbitrary prior distribution P(Xp). We present overview of our approach in Figure 3
In the absence of the target data which the victim CNN classifier is trained, we first train GAN on an arbitrary dataset. Through our regularizer, we encourage the GAN to generate samples from all the modes of the target data. This is achieved by incorporating the pre-trained CNN classifier in the optimization as discussed in Section 3 Once the GAN is trained, we use the generated samples as proxy to the target data for crafting UAPs. Since these samples represent the support of the target data modes, they bring in useful prior about the same, enabling the attacker to craft effective UAPs. In the UAP experiments we use Comics Dataset CGomies-Dataset) as the arbitrary prior P(X,, and use the ResNet-18 (He et al. (2016) classifier trained on ImageNet Deng et al.) (2009) to impart class specific features through the activation maximization loss. However the use Activation maximization (AM) alone with GAN can not encourage GAN to learn features of multiple target classes (i.e. modes). This issue is resolved by making use of our regularizer which encourages the GAN to learn different modes. The final generator objective can then be written as:
Lg Lrel XLreg LAM LAM = Ez-P, [H(C(G(=)))
(10)
(11)
where H(C(G())) the entropy of the classifier output for the generated data. This application is unique to our framework and cannot be realized by other conditional GANs. We use DCGAN architecture to generate 128 X 128 images using prior distribution of comic images (Comics-) Dataset). It is found that (Odena et al. (2017)) generating large number of classes is difficult for a single DCGAN even with conditioning. However, with the proposed regularizer, we are able to generate samples which are classified into very diverse set of 968 ImageNet classes by ResNet- 18 classifier, whereas just using Activation Maximization with GAN resulted in limited set of 25 labels. We also find that diversity in classes helps lot in improving the fooling rate for which ablation results are present in Table The regularizer in each cycle encourages GAN to shift it's focus the underrepresented classes. Due the limited capacity of DCGAN it's bound forget some classes due to shift in focus caused by regularizer. For mitigating this we sample images from multiple cycles the exact details of procedure are described in Appendix A.7 The above procedure was adequate for our experiments, for resolving the forgetting issues in DCGAN large capacity architectures from BigGAN (Brock et al., 2018) can be used. The exact hyperparameters and architecture details are present in the Appendix A.7
We use Generative Adversarial Perturbation (Poursaeed al. 2018) which is an off the shelf algorithm for training generator G for crafting UAPs. We also allow the gradients to flow to deeper ResNet layers using the method introduced by (Wu et al. 2019a). We replace the ImageNet training data with the prior images generated by the GAN described above. We find that a single GAN with ResNet- 18 network is enough to generate effective priors for fooling several ImageNet models. For evaluation we follow existing works and limit the strength of pertur- bation to lx = 10. We report Fooling Rate (FR), which is the percentage of data samples for which addition of UAP flips the predicted label. We log( (C(x,, yx)) (i.e negative of log cross
UAP Generation and Results:
8

---

Under review as a conference paper at ICLR 2021
Method
VGG-16
VGG-19
ResNer-S0
ResNet-152 152
Mean
(Mopuri 2019
64.95
52.49
56,70
44.23
$3.89
Fooling
[Tu et 20199a
70.69
64.98
63.50
46.39
60.69
Rate
AAA
Comiccs
(Mopuri
71.59
72.84
60.72
68.38
GAN AM
63.89
MI-ADy
83.72
Zhung et al 2020
92.20
91.60
79:90
87.9
ImageNer 89.11
Ours
96.16
94.73
83.72
94.00
94.96
Duta"
MI-ADV=
(With ImageNet)
94.30
94.98
90.08
93.12
Table 5: Ablation Different Priors for
ResNet-50 model. *For ImageNet we find that HH (Cr)), Ua) (ie. negative entropy) is more effective hence we report the better fooling rate.
UAP performance (Fooling Rate) the state-of-the-nn art ap- proaches. The Mcan FR is the mean of VGG-16, VGG-19 and ResNet-152 (52 as those are provided by all approaches. These results use MSCOCO distribution which overlups with the target ImageNet categories. "*PT results are with using target ImageNet data itself classilier is trained).
Table 4: Comparison
[2014]
presence
on which
entropy) as the fooling loss as prescribed by (Poursaeed et al. 2018) for all networks. The detailed results presented in Table Note that our data free results better than not only the existing data-free approaches by a large margin but also the recent data-driven method (Zhang et al.] (020) which uses ImageNet training data by considerable 2%. We provide detailed comparison of the data used by various approaches in Appendix A.7
DISCUSSION
In this section we discuss some of the important aspects of our work:
Our approach can be directly applied for semi supervised GAN learning as it decouples classifier
learning and data which can enable learning on unlabeled data.
We would like to emphasize that the presence of classifier in our framework is not disadvan- tage. There has been significant progress classification setup semi supervised learning and learning from long-tailed distributions. We also show that classifiers obtained from such methods can also be used in our framework in Section 4.2
We have noticed while training GAN for the UAP application, on multiple occasions, that texture alone is transferred as a discriminative feature from the classifier. This may be due to the bias of the classifiers towards texture (Geirhos et al.) 2018) and image generation will improve the classifiers improve. However, it still serves as an effective prior about the modes (classes) in the underlying data distribution which the classifiers
trained.
The class balancing problem differs from data coverage problem (Yu et al.) 2020 Srivastava et al.) 2017) as the latter tends make the generated distribution similar to data distribution. Training on long-tale longetailed data can induce the GAN distribution to be long-tailed as well.
CONCLUSION
In this paper, we propose a glass-balancing regularizer balance class distribution of generated samples while training GANs. We present its implications in terms of theoretical bound and com- prehensive experimental analysis in case of loigetailed data distributions. We have demonstrated the utility of our regularizer beyond the GAN framework in crafting input agnostic adversarial per- turbations. The effectiveness of our contribution is exhibited through state-of-the-ar performance on training of GANs long-tailed data distributions well as in crafting Universal Adversarial Perturbations data-free setting.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXtv:1701.07875, 2017.
Wasserstein gan.
arXiv preprint
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXi 1096, 2018.

---

Under review as a conference paper at ICLR 2021
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-disrrbuttionawwer margin loss. In Advances in Neural Information Processing Systems, 2019.
Comics-Dataset. Five thirty eight comic characters dataset. httpttp:: http:://wwwwwwwgl.l.eeee wwggla..gl............kkk....... com/ comic wwwracterrrsseeee
[AVeetairtyeegght/Eiietttttataae ohap dataset Accessed:
2020.
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number samples. In CVPR, 2019.
Harn De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pictquin, and Aaron C Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems, pp. 3594-66024 2017.
Deng, Dong, R. Socher, L.-J. Li. Li, and L. Fei-Fei, ImageNet: Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-rrained cnns biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv: 1..r:1231.11,,,, 2018.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp- 5767-5777, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 2015.
Alexia Jolicoeur- Martineau. The relativistic discriminator: key element missing from standard gan. arXiv preprint 180iv: 00734, 2018
Alexia Jolicoeur-Mart Martineau. On relativistic f-divergences. arXiv preprint arXiv: 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan. John Quan, Tiago Ramalho, Agnieszka Grabska- Grabska-Barwinskk. Barwinska, et al. Overcom- ing catastrophic forgetting neural networks. Proceedings of the national academy of sciences, 114(13):3521-33 -3526, 2017.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. arXiv preprint arXiv:1912.11377 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Karol Kurach, Mario Lučié, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. A large-scale study on regularization and normalization in gans. In International Conference on Machine Learning, pp. 3181-35990 PMLR, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
10

---

Under review as a conference paper at ICLR 2021
Hong Liu, Rongrong Ji, Jie Li, Baochang Zhang, Yue Gao, Yongjian Wu, and Feiyue Huang. Uni- versal adversarial perturbation via prior driven uncertainty approximation. In Proceedings of the IEEE International Conference Computer Vision pp. 2941 2019a.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2019b.
Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. arXiv preprint arXiv: 1903:v:10000277: 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets, 2014.
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In International Conference on Learning Representations, 2018. URL
net fforum?
id=ByS1 VpgRZ.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 100X:00800205557,, 2018.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1765-1773, 2017.
K. R. Mopuri, A. Ganeshan, and R. Venkatesh Babu. Generalizable data-free objective for crafting universal adversarial perturbations. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, (110):2452 2425, 2019.
Konda Reddy Mopuri, Utsav Garg, and R. Venkatesh Babu. Fast feature fool: A data independent approach to universal adversarial perturbations. In Proceedings of the British Machine Vision Conference (BMVC), pp. 30.1-30.12. 2017.
Konda Reddy Mopuri, Utkarsh Ojha, Utsav Garg, and Venkatesh Babu. Nag: Network for adversary generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 742-751, 2018a.
Konda Reddy Mopuri, Phani Krishna Uppala, and R. Venkatesh Babu. Ask, acquire, and attack: Data-free uap generation using class impressions. In Proceedings of the European Conference Computer Vision (ECCV), September 2018b.
Sankha Subhra Mullick, Shounak Datta, and Swagatam Das. Generative adversarial minority over- sampling. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2642-2651. JMLR. org, 2017.
Omid Poursaced, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial pertur- bations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1422-44233 2018
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.064: 2015.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In Advances in Neural Information Processing Systems, pp. 12268-12279, 2019.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Shibani Santurkar, Ludwig Schmidt, and Aleksander Madry. A classification study of covari- ate shift in gan distributions. In International Conference on Machine Learning, pp. 1480-4489. PMLR, 2018.
11

---

Under review as a conference paper at ICLR 2021
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Infor- mation Processing Systems, pp. 1308-33388 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6 2013.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, pp. 7029-7039, 2017.
Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In International Conference Learning Representations, 2019a.
Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXiv preprint arXiv:1912.0095: 2019b.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.03365, 2015. URL http::/dblpp uni-triite de httpdd///ooo///////////aaaaaasat corr/ h0ml1.1..mm Yu2SSX15
Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz. Inclusive gan: Improving data and minority coverage in generative models. arXiv preprint arXiv:2004.0333 03355, 2020.
Chaoning Zhang, Philipp Benz. Tooba Imtiaz, and In So Kweon. Understanding adversarial exam- ples from the mutual influence of images and perturbations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14521-14530, 2020.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. arXiv preprint arXiv:19100 /2027, 2019.
A APPENDIX
A.1 PROOF OF THE PROPOSITION
Proposition: The proposed objective below:
pk log(pe)
max Ph
Σ &
(12)
Ng.
leads to thefollowing bound on pk:
Pk Se - K(lllo())))
N2 En
(13)
where is the number of distinct class labels produced by classifier C.
Proof:
Pk log(pe)
max
Σ
(14)
No.
Introducing the probability constraint and the Lagrange multipltiplier A:
Pr log(pw)
L(P, A)
Σ
X
12 pis 1)
(15)
N
aL Opk
On solving the equations obtained by setting
0:
log (ph
x = 0 - Ps = -1
(16)
N
12

---

Under review as a conference paper at ICLR 2021
aL 0A
Using the constraint
0 get:
Σ Pk -
- =1 -1
Eewi=
(17)
Now normalize both sides by 1/K where K is the distinct labels produced by classifier and apply
Jensen's incquality for concave function v 2 ais;)
2 Eaph(zza and use & as log function:
Σ
2
ANE
A
log
log
K
(18)
K
K
K
K
K
R
On substituting the value of À in incquality:
+ log(ps)
K(1 - log(K) 2 A
Σ N
([g-)o((K))
No
(19)
N
On simplifying and exponentiation we get the following result:
((log(()-11 8i Ex 2%
pi S
(20)
We observe that the penalizing factor K (log(K) 1) is increasing in terms of number of classes K the dataset which is advantageous to us as we need large penalizing factor as Ni/ Σx Ng will be smaller when number of classes is large in the dataset.
A.2 DATASETS
We use CIFAR-I0 (Krizhevsky et al. 2009) dataset for our experiments which has 50k training images and 10k validation images. For the LSUN (Yu al. 2015) dataset use fixed subset of 50k training images for each of bedroom, conference room, dining room, kitchen and living room classes. total have 250k training images and 1.5k validation set of images for LSUN dataset. The imbalanced versions of the datasets are created by removing images from the training set.
A.3 PRE TRAINED CLASSIFIER DETAILS
All the pre-trained classifiers used for Image generation experiments use a ResNet32 et al, 2016) classifier. The classifier is trained using Deferred Re-weighting (DRW) scheme Cao et al. (2019); Cui et al. (2019) with effective number of samples. We use the open source code available https://github.. http:///gthuuuooooaaieeeeee
wwt.waiioc/c.icccccccccccccoccca wtt./kiidc//..AA.. Saarfiic/DDAM--DR DDAM-DRW. We the same learning rate schedule of initial learning rate of 0.01 and multiplying by 0.01 at epoch 160 and 180. We train the models for 200 epochs and start reweighting at epoch 160. We give summary of the validation accuracy of the models in the following table: The classifier obtained by training on the balanced scenario is used as an annotator
Imbalance Ratio
100
10
CIFAR-10
76.67
87.70
92.29
LSUN
82.40
88.07
90.53
Table 6: Validation Accuracy of the PreTrriied Classifiers used with GAN's. The balanced classifier also serves as an annotator.
for obtaining class labels for GAN generated samples. We use the same ResNet32 (He et al. 2016) classifier with the same learning rate schedule as above with cross entropy loss to obtain Classifier Accuracy.
A.4 ARCHITECTURE DETAILS FOR GAN
We the SNDCGAN architecture for experiments CIFAR- 10 and SNResGAN architecture for experiments on LSUN dataset Gulrajani et al. (2017); MMyato et al. (2018). The notation for the ar- [ chitecture tables are follows: m is the batch size, FCldim in, dim out) fully connected Layer,
13

---

Under review as a conference paper at ICLR 2021
CONV(channels..in channels kerneLsize, stride) is convolution layer, TCONV in, channel.our kernel_size, stride) is the transpose convolution layer, BN BatchNorm (loffe & Szegedy 2015) Layer in case of unconditonal GANs and conditional BatchNorm in case of condi- tional GANs. LRelu the leaky relu activation function and GSP is the Global Sum Pooling Layer. Bww.CCCcaaannllls.... chan- The DIS DII._BBICCCChhaaaaeaee BL.OC((hhnneels.tt channels _out, downsampling) and GEN GEN_BBOOC
nels_out, upsampling) correspond to the Discriminator and Generator block used in the (Gulrajani
al. 2017). The architectures are presented in detail in Tables 7 8
and T0
Layer
Input
Output
Operation
Input Layer
(m, 128)
(m, 8192)
FC(128, 8192)
Reshape Layer
(m, 8192)
(m, 4, 4, 512)
RESHAPE
Hidden Layer
(m, 4, 4, 512)
(m, 8, 256)
TCONv(512, 256, 4, 2),BN, TCoNv(256, 128, 4, LL.LLL TCONv(128, 64, 4, BN,
Hidden Layer
(m, 8, 8, 256)
(m, 16, 16, 128)
Hidden Layer
(m, 16, 16, 128)
(m, 32, 32, 64)
Hidden Layer
(m, 32, 32, 64)
(m, 32, 32, 3)
CoNV(64, 3, 3, 1)
Output Layer
(m, 32, 32, 3)
(m, 32, 32, 3)
TANH
Table 7: Generator of SNDCGAN (Miyato et al. 2018 Radford et al. 2015 used for CIFARIO
image synthesis.
Layer
Input
Output
Operation
Input Layer
(m, 32, 32, 3)
(m, 32, 32, 64)
CoNv(3, 64, 3, 1), LRELU
Hidden Layer
(m, 32, 32, 64)
(m, 16, 16, 64)
CoNv(64, 64, 4, 2). LRELU
Hidden Layer
(m, 16, 16, 64)
(m, 16, 16, 128)
Conv(64, 128, 3, 1), LRELU
Hidden Layer
(m, 16, 16, 128)
(m, 8, 8, 128)
CoNv(128, 128, 4, 2), LRELU
Hidden Layer
(m, 8, 8, 128)
(m, 8, 8, 256)
CoNv(128, 256, 3, 1), LRELU
Hidden Layer
(m, 8, 8, 256)
(m, 4, 4, 256)
CoNv(256, 256, 4, 2), LRELU
Hidden Layer
(m, 4, 4, 256)
(m, 4, 4, 512)
CoNv(256, 512, 3, 1), LRELU GSP
Hidden Layer
(m, 4, 4, 512)
(m, 512)
Output Layer
(m, 512)
(m, 1)
FC(512, 1)
Table 8: Discriminator of SNDCGAN
Miyato et al. 2018 used for CIFARI0 image synthesis.
Layer
Input
Output
Operation
Input Layer
(m, 128)
(m, 16384)
FC(128, 16384)
Reshape Layer
(m, 16384)
(m, 4, 4, 1024)
RESHAPE (EN_BLOCK10024, 512, True)
Hidden Layer
(m, 4, 4, 1024)
(m, 8, 8, 512)
Hidden Layer
(m, 8, 8, 512)
(m, 16, 16, 256)
GEN_BLoCK(512, 256, True) GEN.BLOCK(256, 128, True)
Hidden Layer
(m, 16, 16, 256)
(m, 32, 32, 128)
Hidden Layer
(m, 32, 32, 128)
(m, 64, 64, 64)
GEN.BLOCK(12) 64, True)
Hidden Layer
(m, 64, 64, 64)
(m, 64, 64,
BN, RELU, CoNv(64, 3, 1)
Output Layer
(m, 64, 64, 3)
(m, 64, 64, 3)
TANh
Table 9: Generator of SNResGAN used for LSUN image synthesis.
14

---

Under review as a conference paper at ICLR 2021
Layer
Input
Output
Operation
Input Layer
(m, 64, 64, 3)
(m, 32, 32, 64)
DIS BLOCK(3, 64, True)
Hidden Layer
(m, 32, 32, 64)
(m, 16, 16, 128)
Dis_BLocK(64, 128, True) Dis_BLocK(128, 256, True)
Hidden Layer
(m, 16, 16. 128)
(m, 8, 8, 256)
Hidden Layer
(m, 8, 8, 256)
(m, 4, 4, 512)
DIS DI__BLOK((56, 512, True)
Hidden Layer
(m, 4, 4, 512)
(m, 4, 4, 1024)
DIS BLoCK((5122 1024, False), RELU
Hidden Layer
(m, 4, 4, 1024)
(m, 1024)
GSP
Output Layer
(m, 1024)
(m, 1)
FC(1024, 1)
Table 10: Discriminator SNResGAN (Miyato et al. 2018
Gulrajani et al. 2017) used LSUN for
image synthesis.
15

---

Under review as a conference paper at ICLR 2021
Undbl tostions
Figure 4: Effect FID with change in num steps for statistics update(For CIFAR-10 imbalance ratio p = 10)
A.5
HYPERPARAMETER CONFIGURATION (IMAGE GENERATION EXPERIMENTS)
A.5.1
LAMBDA THE REGULARIZER COEFFECIENT
The À hyperparameter is the only hyperparameter that we change across different imbalance scenar- ios. As the overall objective is composed of the two terms:
Lg
- D(z)) XLreg
(21)
As the number of terms in the regularizer objective increase with number of classes K. For making the regularizer term invariant of K and also keeping the scale regularizer term similar to GAN loss, normalize by K. Then the loss is multiplied by A. Hence the effective factor that gets multiplied with regularizer term K
Imbalance Ratio (p)
100
10
CIFAR- 10
10
7.5
LSUN
20
7.5
Table 11: Values of X for different imbalance cases. For LSUN the À gets divide by and for gets divided by 10 before multiplication to regularizer term.
The presence of pre-trained classifier which provides labels for generated images makes it easy to determine the value of A. Although the pre-trained classifier is trained on long-tailed data its label distribution is sufficient provide signal for balance in generated distribution. We the
KL Divergence of labels with respect to uniform distribution for 10k samples in validation stage check for balance in distribution and choose accordingly. We use the FID implementation available here H
A.5.2 OTHER HYPERPARMETERS
We update the effective class distribution periodically after 2k updates (i.e. each cycle defined in section 3 consists of 2k iteration). We find the algorithm performance be stable for large range of update frequency depicted in Figure 4 We also apply Exponential Moving Average on generator weights after 20k steps for better generalization. The hyperparameterss are present in detail in Table I2 Validation Step: We obtain the FID on 10k generated samples after each 2k iterations and choose the checkpoint with best FID for final sampling and FID calulation present in Table Convergence of Network: We find that our GAN + Regularizer setup also achives similar convergence in FID value to the GAN without the regularizer. We show the FID curves for the CIFAR-10 (Imbalance Ratio 10) experiments in Figure
A.6 HYPERPARAMETERS FOR THE SEMI SUPERVISED GAN ARCHITECTURE
We use a ImageNet and ImageNet-21 k pre-trained model with ResNet 50 architecutre as the base model. The fine tuning of the model CIFAR-10 and LSUN has been done by using the code of
16

---

Under review as a conference paper at ICLR 2021
Parameter
Values(CIFAR-II)
Values(LSUN)
Iterations
100k
100k
Generator Ir
0.002
0.002
Discriminator
0.002
0.002
Adam (31
0.5
0.0
Adam (32)
0,999
0,999
Batch Size
256
256
EMA(Start After)
20k
20k
EMA(Decay Rate)
0.9999
0.9999
Table 12: Hyperparameter Setting for Image Generation Experiments.
FID score
90
70
50
30
10
-10
0
40K
100K
60k
100k
(a) GAN
(b) GAN + Proposed Regularizer
Figure 5: Plots of FID (y axis) vs Number of iteration steps. We observe similar curve in both the cases.
notebook present here The accuracy of the classifiers fine-tuned on validation data, trained with 0.1% labelled data (regularizer coeffecient) values are present in the table below:
84.96 % for CIFAR- 10 and 82.40 % for LSUN respectively. The lambda
Imbalance Ratio (p)
100
10
CIFAR-10
10
7.5
LSUN
7.5
Table 13: Values of X for different imbalance cases. For LSUN the À gets divide by 5 and for À gets divided by 10 before multiplication to regularizer term.
The training hyper parameters are same as the ones present in the Table 12 semi supervised experiments we use batch size of 128 fit into GPU memory for semi supervised experiments.
Only in case of LSUN
A.6.1 RESULTS ON CIFAR-100
In this section we show results on CIFAR-100 dataset which has 100 classes having 500 images for each class. We use SNResGAN architecture from Miyaao & Koyama (018)), which is similar to SNResGAN architecure used for LSUN experiments. The architecutre is used for generating 32 x2 images. We use the same hyperparameters used for LSUN experiments listed in Table [2 We use a À value of 0.5 for CIFAR- experimets. The results in Table show that our method on long- tailed CIFARI00 of using GAN Regularizer achieves the best FID and also have class balance similar to cGAN (conditional GAN). The labels for the samples generated by GAN are obtained by a classfier trained on balanced CIFAR- 100 dataset. The KL Divergence between
GAN label distribution and uniform distribution is present in Table 3 The classifier for obtaining class labels for KL Divergence evaluation trained on balanced CIFAR-I01 with setup described in A.3 which serves annotator for all methods.
hhans://gihhaa.ccm/moooorrrrrrre rransterbloimmssccccolabbbiig transfer pytorch.ipynb
17

---

Under review as a conference paper at ICLR 2021
A.7 UAP EXPERIMENTAL DETAILS
(a) Samples from COCO datasct
(b) Samples from Comics Dataset
Figure 6: Comparison of samples used by approach approach by Zhang al. 2020
We use the Comics dataset (Comics- Comics-Datasett Dataset) whereas approach (Zhang et al. 2020) use COCO dataset. COCO dataset has overlap with ImageNet categories. The difference in images used shows that our procedure does not require natural images for generating effective attack. This increases applicability of our method. We use DCGAN architecture generate 128 128 images from the GAN described in Table 15 and [6] In this experiment, we update the mode statistics after The images generated by our method is
Dataset:
every epoch. Hyperparameters are present the table 14 present in figure I0
Parameter
Value 200 epochs (33.6k)
Iterations
Generator Ir
0.002
Discriminator Ir
0.002
Adam (3,)
0.5
Adam (B2)
0.999
À (Regularizer) Batch Size
512
Table 14: Hyperparameters for DCGAN.
Sampling: We find that sampling in different cycles produces samples from diverse classes due to regularizer as it enforces learning of different underrepresented classes in different cycles. Hence we sample 1024 images each from the GAN checkpoints in the last 40 cycles obtain the dataset for UAP generation. This also shows that the regularizer is effective in shifting the distribution of
GAN to produce different modes which is not possible with just Activation Maximization (AM). The increase in number of diverse classes is shown in Figure 7
Generative Adversarial Perturbations: We use the author's Pytorch implementation of the algo- rithm generate attacks. For ResNets allow the gradients to pass through skip connections by using the method of Wu et al. (2019a) with a = 0.5. We train the algorithm for 20 epochs in each case except in case of VGG16. For VGG16 fooling rate converge in 20 epochs.
additional factor of 10 with the loss make
Dodtect Modee (Chos))
702
Cyshe
Figure 7: Number of distinct Modes observed while sampling from different cycles
18

---

Under review as a conference paper at ICLR 2021
Layer
Input
Output
Operation
Input Layer
(m, 100)
(m, 4, 4, 1024)
TCONv(100, 1024, 4, I)
Hidden Layer
(m, 4, 4, 1024)
(m, 8, 8, 512)
TCONV(1024, 512, 4, B,, RELL
Hidden Layer
(m, 8, 8, 512)
(m, 16, 16, 256)
TCoNV(256, 128, 4, 2).BN, R,REL TCoNv(256, 128, 4, B..L TCONv(128, 64, 4, 2),BN,LRELU
Hidden Layer
(m, 16, 16, 256)
(m, 32, 32, 128)
Hidden Layer
(m, 32, 32, 128)
(m, 64, 64, 64)
Hidden Layer
(m, 64, 64, 64)
(m, 128, 128, 3)
TCoNv(64, LL,,N,
Output Layer
(m, 128, 128, 3)
(m, 128, 128,
TANH
Table 15: Generator of DCGAN (Radford et al.
2015) used for UAP Experiments.
Layer
Input
Output
Operation
Input Layer
(m, 128, 128, 3)
(m, 64, 64, 64)
CoNv(3, 64, 4, 2), LRELU
Hidden Layer
(m, 64, 64, 64)
(m, 32, 32, 128)
CoNv(64, 128, 4, 2),BN, LRELU
Hidden Layer
(m, 32, 32, 128)
(m, 16, 16, 256)
CONv(128, 256, 4, 2),BN, LRELU CoNv(256, 512, 4, 2),BN, LRELU
Hidden Layer
(m, 16, 16, 256)
(m, 8, 8, 512)
Hidden Layer
(m, 8, 8, 512)
(m, 4, 4, 1024)
CoNv(512, 1024, 4, 2),BN, LRELU
Output Layer
(m, 4, 4, 1024)
(m, 1)
CoNv(1024, 1, 4, 1), LRELU
Table 16: Discriminator of DCGAN (Radford et al.
2015 used for UAP Experiments.
(a) ACGAN (Conditional)
(b) cGAN (Conditional)
(c) SNResGAN (Unconditional)
(d) Ours (Unconditional)
Figure 8: Images from different GANs with imbalance ratio (p 10)
19

---

Under review as a conference paper at ICLR 2021
(a) ACGAN (Conditional)
(b) cGAN (Conditional)
(c) SNDCGAN (Unconditional)
(d) Ours (Unconditional)
Figure 9: Images generated by different GANs for CIFAR- 10 with imbalance ratio (p = 10).
20

---

Under review as a conference paper at ICLR 2021
Figure 10: Images generated for CIFAR-100 dataset with our method (GAN + Regularizer).
21


Reviews:
[{"review_id": "ohBUTWW1pb-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the problem that GANs' poor performance in the imbalanced dataset and presents the class balancing regularizer for training GANs, encouraging the GAN to pay more attention to underrepresented classes. \nThey induced the class distribution information using a pre-trained classifier, and the regularize utilizes the class distribution to penalize excessive generation of samples from the majority classes, thus enforcing the GAN to generate samples from minority classes. \n\nPros:\n+ The motivation is clear.\n+ It seems to cite the relevant literature (that I know of) and compare it to reasonably established attacks and defenses.\n+ Simple/directly applicable approach that seems to work experimentally, but\n\nCons:\n- The method part is not easy to follow. My understanding is the effective class frequency is the cumulative number of generated samples for each class (with a discount factor) and normalizing it yields the distribution of the generated samples. But I didn't get the relationship between this distribution and the following regularizer. Could you comment on that?\n- As far as I understand, the regularizer encourages all the classes to be balanced within each batch by maximizing the entropy. I am a little bit concerned about this setting, what if we use a small batch size so that the class distribution can be imbalanced within one batch.\n- Only LSUN subset and CIFAR10, which include 5 and 10 classes respectively. I am wondering about the performance on large-scale imbalanced datasets like iNaturalist.\n\n\n----\nUpdated:\nThanks to the authors for the provided extra experiments and clarifications. Some of my concerns (e.g., how the batch size affected the performance) have been diminished, but I do agree with other reviewers that more baselines should be included. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "hk7AFnAt9dS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\n\nThe paper proposes a regularizer to force an unconditional GAN generator to produce samples that follow a uniform class distribution. To provide feedback to the generator about the class distribution over the generated images, the proposed method utilizes a pretrained classifier on the same (imbalanced) training dataset. Motivated by the exponential forgetting of earlier tasks in neural networks [1], the regularization term encourages the generator to increase the proportion of samples of an infrequent class after a certain number of iterations and vice versa. Empirical studies are performed to show the effectiveness of the regularization: 1) the paper shows that the proposed method enables generating samples with a uniform class distribution with a GAN trained on a dataset with a long-tailed class distribution and (2) that the method benefits in generating universal adversarial perturbations (UAPs) in the data-free scenario. \n\nPros:\n\n1.\tThe paper studies an important and challenging problem of training GANs on imbalanced dataset.\n2.\tThe proposed regularization term is novel, the derivation of the regularization term is well explained.\n\nCons:\n\n1. Image Generation\n\n-\tIt is not clear whether it’s necessary to obtain a set of balanced samples in such a complicated way. Given a pretrained classifier as used in the proposed method, we could simply use the classifier to select samples after training a standard unconditional GAN. This simple baseline experiment is missing in the paper.\n\n-\tThe experimental setup in Section 4.1 might be unfair for the unconditional GAN baselines. If the model is trained on an imbalanced dataset and tested on a balanced dataset, then the comparison between the proposed method and unconditional GANs is not fair. FID for the latter might be worse simply because the training and test distributions are different, whereas the proposed method is tailored for this special purpose. \n\n-\tThe paper highlights that the method can generate images with uniform class distribution even in the unconditional case, where no labels are given to the generator. This would be useful if the classifier training is decoupled from the training data of the generator (this is not the case in the presented experiments, as the classifier uses the same training data as the generator with GT class labels). E.g. the classifier is trained on one dataset and then transferred for GAN training on another similar but unlabelled dataset, producing sharp images, or if only part of the training data was labelled, thus reducing the need for labels. Such experiments would help to support the claims in the paper. After all, the method is quite interesting, since in contrast to a conventional conditional GAN, the discriminator is not provided with the class identity of the generated image directly. \n\n- The method and evaluation heavily relies on the quality of the pre-trained classifier. In the presented experimental setup, the classifier is trained on the same training set as the GAN model. So in case of the highly imbalanced training set, it's not clear how well the classifier can recognize the imbalanced classes. Thus the proposed model might suffer from the same problem as the unconditional GAN if the classifier has troubles recognizing imbalanced classes and is biased towards well represented classes.\n\n- The problem posed in the paper is that unconditional GANs do not sample images from the present classes uniformly, but are biased by the class distribution in the dataset. This makes sense in the unconditional case, but Figure 2 shows that this also happens in the conditional case for ACGAN. Some further explanation for why that is would be helpful. Figure 2 is not discussed in the text, but it should be. \n\n-\tThe produced samples are uniformly classified into different categories by the classifier. It would be good to provide a figure with generated images and the corresponding predicted labels, to see if the prediction and actual content match as well as in the case of a conditional GAN, like cGAN.\n\n-\tThe proposed method is also trained with a big batch size of 256, which is very helpful for covering all classes. It would be more useful to see if the method also works well with small batch sizes of 16 or 32, which are common for high resolution GAN image synthesis.\n\n-\tBoth the conditional GANs and the proposed regularizer use the same labelled data.\n\n-\tThe paper says that in the highly imbalanced case cGAN sufferes from the training instability. In this case how is the batch formed for cGAN training? Do you balance the batch in terms of classes (as the batch size of 256 is quite large)? If not, it would be interesting to see how cGAN performs with the balanced batch.\n\n2. Universal Adversarial Perturbations\n\n-\tThe experimental results in Section 4.2 does not look convincing. The listed methods have different degrees of available information (either overlaps with the target dataset or a pretrained classifier on the target dataset). Moreover, for the proposed method, the requirement of a pretrained classifier on the target dataset is a strong limitation.\n\n-\tThe paper claims that the method can help generate UAPs in the absence of the target dataset for which a classifier shall be fooled. The target dataset is ImageNet. The dataset on which the auxiliary classifier for the proposed regularization loss is trained is also ImageNet. Hence, it seems the UAPs could have been learned from ImageNet directly. To avoid confusion, sentences like the following need more elaboration: \"We also find that our data free results are at par with the recently published method (Zhang et al., 2020) which uses ImageNet training data.\"\n\n-\tThe following sentence needs more explanation: \"Our approach achieves diversity through sampling from multiple checkpoints, as in each cycle the regularizer encourages the GAN to focus on different poorly represented classes\". If checkpoints from different training steps are used, it seems that the target distribution is not really captured uniformly, contrary to the aim of the paper. Hence, it would be good to add some explanation here to prevent misunderstandings.\n\n-\tTable 2 is not discussed in the text.\n\nMinor comments:\n\n-  The height-width ratio of Figure 1b should be rectified.\n-  The discussion is not well written. Especially the message of the 2nd bullet point needs more clarifications.\n\n\nReview summary:\n\nThe paper shows that the classification and image generation can be decoupled for GANs. This makes the setup semi-supervised, not unconditional (as claimed in the paper). While this is interesting, it would be good for the paper to show an application where this is actually useful, because in all provided examples the ground truth labels of the training set are present and used for training the classifier, but are just not used directly to train the discriminator. For example, when it comes to UAPs it is not clear to what extent the proposed approach is data-free if ImageNet is used for training the regularizer as well as the classifier to be fooled. Thus, the main weaknesses of the paper in my opinion are the significance of the presented results, unfairness in the experimental setup, and the clarity of presentation. \n\n\nPost-rebuttal feedback:\n\nThanks to the authors for the provided extra experiments and clarifications. I feel that my concerns have been partially addressed, thus raising my score to 5. I still think that the proposed method is limited by the classifier, it's ability to capture a long-tailed distribution, which is not so easy to get when trained on imbalanced dataset. This significantly limits the applications of the proposed approach in real-life scenarios. The paper has also experimented only on artificially created imbalanced datasets, which contain small number of classes with the model being trained with the batch size higher than the number of classes. It would be beneficial to see how the model would perform in more realistic setup when the number of classes is significantly bigger than the batch size (e.g. iNaturalist or even ImageNet), to support more the claims of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fm9JEvNOmHX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Overview**: The paper presents a simple regularizer term that aims to force a GAN to generate samples following a uniform distribution over different classes. The regularizer depends on a classifier that works well on an imbalanced or long-tailed dataset. The paper presents experiments on CIFAR-10 and LSUN that were synthetically long-tailed or imbalanced. The results show that the proposed term generates samples that follow a more uniform distribution over classes.\n\n*Pros*:\n- Interesting idea as it can help a generative algorithm to remove an imbalance from a dataset. \n- The proposed regularizer is simple but depends on a classifier (see below for more details).\n\n*Cons*:\n- The regularization term depends on a classifier that works well already on the imbalanced dataset. Getting a classifier to work on long-tailed datasets is not an easy task and people are still investigating the development of techniques to learn from imbalanced datasets (see for example I).  From a practical point of view, this is a hard requirement that can reduce the chances of adoption.\n\n- Proposed loss may have conflicting terms. The final loss composed of the relativistic loss and the regularizer may be conflicting. According to the text (below Eq. 3), this loss follows the training distribution which in the context of the paper is long-tailed. However, the proposed regularizer penalizes the GAN to generate samples following a long-tailed distribution. Aren't these two terms then conflicting? If so, can this conflict impact the convergence of the network?\n\n- Insufficient experiments. While the experiments show good results on two small and synthetically long-tailed datasets, it is unclear if this method can work on naturally long-tailed datasets (e.g., iNaturalist). Unfortunately, the CIFAR-10 and LSUN datasets have a small set of classes in them. How does this method work on naturally long-tailed (e.g., iNaturalist) and/or large-scale datasets with a larger set of classes (e.g., ImageNet-LT)? Also, how do the generated images look like? Does this method still preserve a good perceptual image?\n\n- Lack of clear impact on applications. After reading the introduction, I did not have a clear application where this component can be crucial to either enable a new application or solve a bottleneck. The discussion section briefly mentions a few applications. However, I think the paper would've been stronger if it showed experiments using the proposed approach and showing its impact on a clear application.\n\nReferences:\nI. Liu et al. Large-Scale Long-Tailed Recognition in an Open World. CVPR 2019.\n\nMinor comments:\n1. The contribution list of the Introduction section uses terms that have not been defined, i.e., FID and UAP.\n2. If using latex, please use \\min, \\max, \\log to properly display the operators.\n\n----------------------------------------------------\nPost Rebuttal Update\n\nWhile I think the idea is interesting, I still think the proposed loss is not consistent as I still think the two terms in the loss collide with each other, its practical value is limited mainly because making a GAN to work on various datasets is a challenging task, and that the experiments now raised more questions than answers. For these reasons I still lean towards rejection as I believe the paper can benefit from a revision.", "extra_scores": {}, "extra_text": {}}, {"review_id": "C-0hAOsZ6Ch", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper tries to solve the data imbalance problem in conditional GAN by adding a classification loss as the constraint. This constraint can be seen as weighted softmax and the weight is the smoothed number of classes showed before. \n\nThis paper is well written and structured. The idea is intuitive. The authors applied the methods in long-tail classification to GAN. In the experiment part, the authors first did the image generation from long-tailed distribution on CIFAR10 and LSUN, the apply this technique to data-free universal adversarial perturbation.  The results are better than the ones without the proposed constraints.\n\nMy main concern is that the dataset used in the first part is too simple to reflect the strength of the algorithm. Since the method is very intuitive, more experiments on datasets with larger pixel number and more categories are more convincing. \n\nAnother concern is the baseline. Since current state-of-art conditional GAN can reach lower FID (compare with FID score in the last column of Table 1), it might be better to compare with those methods. And there exist other intuitive baselines like data-augmentation and resampling.  is it fair to compare your methods with these?\n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have provided very detailed responses and added additional experimental results, which have helped address some of the referees' concerns. However, since the modification made to a vanilla GAN algorithm is relatively small, the reviewers are hoping to see the experiments on more appropriate real-world datasets (not artificially created imbalanced datasets with relatively few classes), more/stronger baselines, and rigorous theoretical/empirical analysis of the method's sensitivity to the quality of the pre-trained classifier. The paper is not ready for publication without these improvements. 
Decision: Reject