Stein Latent Optimization for Generative Adversarial Networks
Authors: Uiwon Hwang
Heeseung Kim
Dahuin Jung
Hyemi Jang
Hyungyu Lee
Sungroh Yoon
Venue: ICLR 2022 Poster (2022)
Keywords: Generative Adversarial Networks
Unsupervised Conditional GANs

Abstract: Generative adversarial networks (GANs) with clustered latent spaces can perform conditional generation in a completely unsupervised manner. In the real world, the salient attributes of unlabeled data can be imbalanced. However, most of existing unsupervised conditional GANs cannot cluster attributes of these data in their latent spaces properly because they assume uniform distributions of the attributes. To address this problem, we theoretically derive Stein latent optimization that provides reparameterizable gradient estimations of the latent distribution parameters assuming a Gaussian mixture prior in a continuous latent space. Structurally, we introduce an encoder network and novel unsupervised conditional contrastive loss to ensure that data generated from a single mixture component represent a single attribute. We confirm that the proposed method, named Stein Latent Optimization for GANs (SLOGAN), successfully learns balanced or imbalanced attributes and achieves state-of-the-art unsupervised conditional generation performance even in the absence of attribute information (e.g., the imbalance ratio). Moreover, we demonstrate that the attributes to be learned can be manipulated using a small amount of probe data.

Published as a conference paper at ICLR 2022
STEIN LATENT OPTIMIZATION FOR GENERATIVE ADVERSARIAL NETWORKS
Uiwon Hwang', Heeseung Kim' Dahuin Jung', Hyemi Jang', HHyunyu Lee', Sungroh Yoon 1.2, "Department of Electrical and Computer Engineering, *AIIS, ASRI, INMC, ISRC, NSI, and Interdisciplinary Program Artificial Intelligence, Scoul National University, Scoul 08826, Korea {ui len. hwang, gmd789,, anna annajung0 625, wkdal 9512, rucy74, sryoon 8 8nnu ac. kr
ABSTRACT
Generative adversarial networks (GANs) with clustered latent spaces can perform conditional generation in a completely unsupervised manner. In the real world, the salient attributes of unlabeled data can be imbalanced. However, most of existing unsupervised conditional GANs cannot cluster attributes of these data in their latent spaces properly because they assume uniform distributions of the attributes. To address this problem, we theoretically derive Stein latent optimization that pro- vides reparameterizable gradient estimations of the latent distribution parameters assuming a Gaussian mixture prior in continuous latent space. Structurally, we introduce an encoder network and novel unsupervised conditional contrastive loss ensure that data generated from single mixture component represent single attribute. We confirm that the proposed method, named Stein Latent Optimization for GANs (SLOGAN), successfully learns balanced imbalanced attributes and achieves state-of-the-art unsupervised conditional generation performance even in the absence of attribute information (e.g.. the imbalance ratio). Moreover, we demonstrate that the attributes to be learned can be manipulated using a small amount of probe data.
INTRODUCTION
GANs have shown remarkable results in the synthesis of realistic data conditioned on a specific class (Odena et al. 2017: Miyato & Koyama| 2018 Kang & Park (020). Training conditional GANs requires massive amount of labeled data; however, data are often unlabeled or possess only a few labels. For unsupervised conditional generation, the salient attributes of the data are first identified by unsupervised learning and used for conditional generation of data. Recently, several unsupervised conditional GANs have been proposed (Chen al. 2016 Mukherjee al., 2019 Pan al. 202T, Armandpour et al. 2021). By maximizing lower bound of mutual information between latent codes and generated data, they cluster the attributes of the underlying data distribution their latent spaces. These GANs achieve satisfactory performance when the salient attributes of data are balanced.
However, the attributes of real-world data can be imbalanced. For example, the CelebA dataset (Liu et al. 2015), examples with one attribute (not wearing eyeglasses) outnumber the other attribute (wearing cyeglasses). Similarly, the number of examples with disease-related attributes in a biomed- ical dataset might be miniscule (Hwang al. (019)). Thus. the imbalanced nature of real-world attributes must be considered for unsupervised conditional generation. Most of cxisting unsupervised conditional GANs are not suitable for real-world attributes, because they assume balanced attributes if the imbalance ratio is unknown (Chen et al. (016)] Mukherjee et al.) 2019: Pan et al., 2021). Examples where existing methods fail to learn imbalanced attributes are shown in Figure | (a), (b) and (c).
In this paper, we propose unsupervised conditional GANs, referred to as Stein Latent Optimization for GANs (SLOGAN). We define the latent distribution of the GAN models Gaussian mixtures to enable the imbalanced attributes to be naturally clustered in a continuous latent space. We derive reparamcterizable gradient identities for the mean vectors, full covariance matrices, and mixing coefficients of the latent distribution using Stein's 's lemma. This enables stable learning and makes
"Corresponding author

---

Published as a conference paper at ICLR 2022
(a) InfoGAN
(b) DeLiGAN
(c) ClustaGAN
(d) SLOGAN (Ours)
Figure 1: Unsupervised conditional generation on synthetic dataset. Dataset consists eight two- dimensional Gaussians (gray dots), and the number of unlabeled data instances from each Gaussian distribution is imbalanced (clockwise from the top. imbalance ratio between the first four Gaussians and the remaining four is 1 1:3). It is considered that the instances sampled from the same Gaussian share an attribute. Dots with different colors denote the data generated from different latent codes. Bold circles represent the samples generated from the mean vectors of latent distributions.
latent distribution parameters, including the mixing coefficient, learnable. We then devise a GAN framework with an encoder network and unsupervised conditional contrastive loss (U2C loss), which can interact well with the learnable Gaussian mixture prior (Figure 2 This framework facilitates the association of data generated from Gaussian component with single attribute.
For the synthetic dataset, our method (Figure (d) shows superior performance on unsupervised conditional generation, with the accurately learned mixing coefficients. We performed experiments
various real-world datasets including MNIST (LeCun et al.
1998), Fashion-MNIST (Xiao et al. 2017
(IFAR-10 (Krizhevsky al., 2009), CelebA (Liu al.
2015). CelebA-HQ (Karras al. 2017).
and AFHQ (Choi et al. 2020) using architectures such as DCGAN (Radford et al. (016)), ResGAN
verified that the proposed method outperforms existing unsupervised conditional GANs in unsupervised conditional generation on datasets with balanced or imbalanced attributes. Furthermore, we confirmed that we could control the attributes to be learned when a small set of probe data is provided.
(Gulrajani et al.) 2017) and StyleGAN2 (Karras al. 2020). Through experiments,
The contributions of this work are summarized as follows:
We propose novel Stein Latent Optimization for GANs (SLOGAN). To the best of our knowledge, this is one of the first methods that can perform unsupervised conditional generation by considering the imbalanced attributes of real-world data.
To enable this, derive the implicit reparameterization for Gaussian mixture prior using Stein's 's lemma. Then, we devise GAN framework with an encoder and an unsupervised conditional contrastive loss (U2C loss) suitable for implicit reparameterization.
SLOGAN significantly outperforms the existing methods on unsupervised learning tasks, such as cluster assignment, unconditional data generation, and unsupervised conditional generation, on datasets that include balanced or imbalanced attributes.
2 BACKGROUND
2.1 GENERATIVE ADVERSARIAL NETWORKS
Unsupervised conditional generation Several models including InfoGAN (Chen al. 2016).
ClusterGAN Mukkherjee et al. 2211)). Self conditiondt GAN (Liu et al. 2020). CD-GAN (Pan et al. 202Tp, and PGMGAN (Armandpour et al. 2021 have been proposed to perform conditional genera- tion in a completely unsupervised manner. However, these models primarily have two drawbacks: (1) Most of these methods embed the attributes in discrete variables, which induces discontinuity among the embedded attributes. (2) Most of them assume uniform distributions of the attributes, and thus fail to learn the imbalance in attributes when the imbalance ratio is not provided. In Appendix B.2 we discuss the above models in detail. By contrast, work addresses the aforementioned limitations by combining GANs with the gradient estimation of the Gaussian mixture prior via Stein's 's lemma and representation learning the latent space.
2

---

Published as a conference paper at ICLR 2022
(tfc)
Gaussian prior ((x: p) (Sec. 3.1)
Z
G
Xy
c= k
Mx
Mr
Rc z
MC
(Sec. 3.3)
D
dx
Implicit reporameterization
(Sec. 3.2)
cosé
VuC VzC
PpC
ex
X
Adversarial loss
Unsupervised conditional contrastive loss (U2C loss)
Figure 2: Overview the SLOGAN model. Here, X, denotes the data generated from latent vector is real data that is used for adversarial learning, and C indicates component ID of the Gaussian mixture prior with the highest responsibility argmax, ((clz).
GANs with Gaussian mixture prior DeLiGAN (Gurumurthy et al. 2017) analogous to the proposed method, as it assumes a Gaussian mixture prior and learns the mean vectors and covariance matrices via the reparameterization trick. However, DeLiGAN assumes uniform mixing coefficients without updating them. As result, it fails perform unsupervised conditional generation datasets with imbalanced attributes. In addition, it uses the explicit reparameterization trick, which inevitably suffers from high variance in the estimated gradients. This will be discussed further in Section 2.3
2.2 CONTRASTIVE LEARNING
Contrastive learning aims to learn representations by contrasting neighboring with non-neighboring instances (Hadsell et al., 2006). In general, contrastive loss is defined as critic function that approximates the log density ratio log p(yl=)/p(() of two random variables X and Y. By minimizing the loss, the lower bound 2019). Several studies have shown that contrastive losses are advantageous for the representation learning of imbalanced data (Kang et al.) 2021 2020 Wanyya et al. 202T). Motivated by these observations, we propose contrastive loss that cooperates with a learnable latent distribution.
mutual information / ((X; Y) is approximately maximized (Poole et al.,
2.3 GRADIENT ESTIMATION FOR GAUSSIAN MIXTURE
Stein's lemma Stein's ' lemma provides a first-order gradient identity for multivariate Gaussian distribution. The univariate case of Stein's 'ss lemma be described as follows:
Lemma 1, Let function i((.) R + be continuously differentiable. q(2) is univariate Gaussian
distribution parameterized by the mean μ and variance Then, the following identity holds:
Eq(=) [o- (-))\(22))) Ex((2) [Vz((2)]
Lin et al. (2019b) generalized Stein's 'si's lemma to exponential family mixtures and linked it to the
implicit reparameterization trick. Stein's 's lemma has been applied to various fields of deep learning, including Bayesian deep learning (Lin et al., (219a) and adversarial robustness (Wang et al.) 2020). To the best of our knowledge, our work is the first to apply Stein's 's lemma to GANs.
A simple method estimate gradients of the parameters of Gaussian mixtures is explicit reparameterization, used DeLiGAN. When the c-th component is selected according to the mixing coefficient p(c), the latent variable is calculated follows: :4 + >
Reparameterization trick
where Ao I). Derivatives of loss function 0C(=) and @C(=) "E only update the mean and covariance matrices the selected (c-th) component μ4. and E., respectively. Gradient estimation using explicit reparameterization is unbiased; however, it has a distinctly high variance. For single latent vector z, the implicit reparameterization trick (Figurnov et al. 2018) updates the parameters of all the latent components. Gradient estimation using implicit reparameterization is unbiased and has a lower variance, which enables a more stable and faster convergence of the model. The gradients for the parameters of the Gaussian mixture prior in method implicitly reparameterizable.
øp
3

---

Published as a conference paper at ICLR 2022
3 PROPOSED METHOD
In the following sections, we propose Stein Latent Optimization for GANs (SLOGAN). We assume Gaussian mixture prior (Section 3.1), derive implicit reparameterization of the parameters of the mixture prior (Section (.2). and construct a GAN framework with U2C loss (Section (33). Additionally, we devise method manipulate attributes to be learned if necessary (Section 3.4 overview of SLOGAN is shown in Figure
3.1 GAUSSIAN MIXTURE PRIOR
We consider a GAN with generator G Rd. + Rd, and discriminator D RRda + R. where d₂ and d, are the dimensions of latent and data spaces, respectively. In the latent space 2 R we consider a conditional latent distribution q(z/c) (2: fee Ee), c 1,.., K, where K is the number of components we initially set and pe: E. the mean vector and covariance matrix of the th component, respectively. Subsequently, we consider a Gaussian mixture q(z) Ee p(c)g(zle) parameterized by μ [ Σ [Ee] and π { {p(c)) as the prior.
We hypothesize that a mixture prior in a continuous space could model some continuous attributes of real-world data (c.g.. hair color) more naturally than categorical priors which could introduce discontinuity (Mukherjee et al.) 2019). Because we use implicit reparameterization of a mixture of Gaussian priors (derived in Section 3.2), SLOGAN can fully benefit from implicit reparameterization and U2C loss. By contrast, the implicit reparameterization of prior distributions that do not belong to the exponential family (e.g., categorical priors) remains an open question.
In the experiments, the elements of μ.. were sampled from (0, 0.1), and we selected E, = / and Te = 1/K as the initial values. For the convenience of notation, we define the latent distribution q (z). the mixing coefficient ne p(c). and 8(z) {o(z). ) c= where 8(z)c q(z c)
((clz), the responsibility of component for a latent vector z, can be expressed as follows:
q(clz) = q(c,z) g(zlc)p(c) 5(z)eTe
(2)
(()
q(z)
3.2 GRADIENT IDENTITIES
We present gradient identities for the latent distribution parameters. To derive the identities, the generalized Stein's 's lemma for Gaussian mixtures with full covariance matrices (Lin et al. 2019b).
First, we derive a gradient identity for the mean vector using Bonnet's s theorem (Bonnel 1964..
Theorem 1. Given an expected loss of the generator L and a loss function for sample ((-) Rd
R. we assume & to be continuously differentiable. Then, the following identity holds:
Vp, Eg (0(z)==.
(3)
Proof of Theorem iis given in Appendix C.I
We derive a gradient identity for the covariance matrix via Price's theorem (Price 1958). Among the two versions of the Price's 's theorem, use the first-order identity minimize computational cost.
Theorem 2. With the same assumptions as in Theorem !
the following gradient identity holds:
V C - _ [8(z).T.E= pe) V [())]
Proof of Theorem given in Appendix C.2 In the implementation,
replaced the expectation of the right-hand side of Equation 4 with the average for batch of latent vectors; hence, the updated E may not be symmetric or positive-definit. To force a valid covariance matrix, we modify the updates of the covariance matrix as follows:
1 AE -V,C
=
1
(Se
st)
(5)
AE = AE, + AAEDD-AAE
(6)
4

---

Published as a conference paper at ICLR 2022
Algorithm 1 Training procedure of SLOGAN
Algorithm 2 Intra-cluster FID
Initialize μr, E. P. parameters of D, G, and E
input [[{x]] 1) Data sampled from
while training loss not converged do
p(x for y 1, ...... K;
Sample batch of data [x') [x') ~>p(x)
] 1==1 jKk Latent vectors sam-
Sample batch of latent vectors {z ) g(z)
pled from ((((c) for I,... K
for I,..., B do
output: ICFID Intra-cluster FID;
Calculate ladv(z') (z') and (zzc(z²) for latent vector x² Szi 4 --0(2))xo02 / (xx'=(x)) - (
Y. Class-ccclute assignments
Y [1. 1..... K}
end for
C (1,.., K]
for c L,.., K do
for each class in Y do
Update pc. Er and p. via stochastic gradient estimation
X. s- ]im]
Me t -a B Σi= 5(x5 dem") (twa. (z') (A()+a02))))")
for each cluster in C' do 1x
AΣ. + 4B ΣE= (Sa +
ST)
X,
Ee < Le 7 (A JAE AE.)
d(y. c) + (r, X,) end for
P: t Pc - =V Di=1 To (ó(z"). 1) Endv (z')
c² +- argmin occ d(y. e) ICFID d(y.c')
end for
Update G. E' and D using SGD
Y.(y)
VG,E
1 Σ (fadr (z
B
Remove e² from C end for Σ ICFID())
Vn Σ Di=l ii= lavv (2') B Ei= D(x'))
ICHID
end while
where S2 = TT (z- p.) V [(z). and 7 denotes the learning rate for Σ... Equation holds as AΣ, = 1/2 Eg [S = EE 7 Motivated by Lin et al. (020)) Equation ensures the defiitteedss of the covariance matrix, which is proved by Theorem 3
Theorem 3. The updated covariance matrix Σ. = E. + with the modified update rule specified in Equation iss positive-definte if E.e is positive-defmite.
Proof of Theorem is provided in Appendix C.3
We introduce mixing coefficient parameter Pc. which is updated instead of the mixing coefficient Te, to guarantee that the updated mixing coefficients are non-negative and summed to one. Te can be calculated using the softmax function (i.e., Te exp(pe)/ Σ exp(p:)). We can then derive the gradient identity for the mixing coefficient parameter as follows:
Theorem 4. Let Pe be a mixing coefficient parameter. Then, the following gradient identity holds:
V pc L = Eg [ (5(z). - 1) E())
Proof of Theorem is given in Appendix C.4 Because the gradients of the latent vector with respect to the latent parameters are computed by implicit differentiation via Stein's lemma, obtain the implicit reparameterizzatio gradients introduced by Figurnov et al. (2018).
3.3 CONTRASTIVE LEARNING
We introduce new unsupervised conditional contrastive loss (U2C loss) to learn salient attributes from data and to facilitate unsupervised conditional generation. We consider a batch of latent vectors (z)) { q(z), where is the batch size. Generator G receives the i-th latent vector z' and generates data x = G(z²). The adversarial loss for G with respect to the sample z is as follows:
Ladv = -DG((z)))
We also introduce an encoder network E implement U2C loss. The synthesized data xJ. enters E, and generates an encoded vector e = E(x)). Then, we find the mean vector p where C is the component ID with the highest responsibility ((clz²). We calculate C first because a generated sample should have the attribute of the most responsible component among multiple components in the continuous space. Second, to update the parameters of the prior using implicit reparameterization, the loss should function of latent vector z', as proved in Theorems 1 and 4 ID for each sample is calculated as follows:
The component
C argmax q(clz') = argmax 8(zz) )eme
(9)
5

---

Published as a conference paper at ICLR 2022
where (<clz') 0(
_ is derived from Equation
To satisfy the assumption of the continuously and we adopt the Gumbel-Softuax relaxation (Jang
2
differentiable loss function in Theorems 1
et al. 2017 instead of the argmax function. We use f'c = > C'ule to calculate U2C loss to ensure that the loss function is continuously differentiable with respect to z', where C Gunbel-Sollmax, (T) and 0.01. We derive U2C loss as follows:
exp(cos O(s)
(Uzc(z') = log
(10)
B Σニ= exp(cos 0ij)
where we select the cosine similarity between ex and ulg cos 0,j = _ /C//le I| as the critic function that approximates the log density ratio log p(C') (1(')x///////// (C')) for contrastive learning. Given test data, the probability for each cluster can be calculated using the assumption of the critic function, which enables us assign a cluster for the data. Cluster assignment is described in Appendix D.2
Intuitively, mean vector HC of a latent mixture component is regarded as a prototype of each attribute. U2C loss encourages the encoded vector ex of the generated sample to be similar to its assigned low-dimensional prototype Mec in the latent space. This allows each salient attribute clusters in the latent space, and each component of the learned latent distribution is responsible for a certain attribute of the data. If cos Og proportional to the log density ratio log p( C*
minimizing U2C loss in Equation 10 is equivalent to maximizing the lower bound of the mutual information I(C x,), as discussed by Poole al. (2019) and Zhong et al. (2020).
G C'and E are trained to minimize 1/2 Eé= (fadv (z') + Al(2o(z))), )), where À denotes the coefficient of U2C loss. Both ya and E are learned by substituting (adv(z') (2') into l of Equations 3 and respectively. When U2C loss is used to update æ, U2C loss hinders ı from estimating the imbalance ratio of attributes in the data well, which is discussed in Appendix [A.3 with detailed explanation and an empirical result. Therefore, p. from which æ is calculated, uses only the adversarial loss, and é of Equation 7 is substituted by Ladv (z'). fs, Σ and p are learned using a batch average of estimated gradients, which is referred to as stochastic gradient estimation, instead of expectation over the latent distribution q. The entire training procedure of SLOGAN is presented in Algorithm !
To help that the latent space does not learn low-level attributes, such as background color, we additionally used the SimCLR (Chen et al. 2020) loss on the generated data with DiffAugment (Zhao et al.; 2020) to train the encoder on colored image datasets. Methodological details and discussion on SimCLR are presented in Appendix D.4 and A.3 respectively.
3.4 ATTRIBUTE MANIPULATION
For datascts such face attributes, data point can have multiple attributes simultancously. To learn M iim for the c-th latent component, which consists of M data points with the desired attribute, can be utilized. We propose the following loss:
a desired attribute from such data, a probe dataset [
M
exp(cos 0%)
Lm
log
(11)
M
Σ exp(cos of.)
where cos 0% = Fi(x) /=/l| E(x) (xx) 11 is the cosine similarity between Fx(:)) and μ Our model manipulates attributes by minimizing f. for μs. E, G, and F.. In addition, mixup (Zhang et 2018) can be used to better learn attributes from a small probe dataset. The advantage of SLOGAN in attribute manipulation is that it can learn imbalanced attributes even the attributes in the probe dataset are balanced, and perform better conditional generation. The detailed procedure of attribute manipulation is described Appendix D.3
4 EXPERIMENTS
4.1
DATASETS
We used the MNIST (LeCun et al. 1998). Fashion- (FMNIST) (Xiao al. 2017). CIFAR-10
Krizhevsky et al. 2009), CelebA (Liu et al., 2015), CelebA-HQ (Karras et al. 2017). and AFHQ
Choi al. 2020) datasets evaluate the proposed method. We also constructed some datasets with

---

Published as a conference paper at ICLR 2022
Table 1: Performance comparison on balanced attributes
DeLiGAN
DeLiGGAN+
ClusterGIAN
SCGAN
CD-GAN
PGMGAN
SLOGAN
NML
0.64+0002
0.64+0.03
0.57=0.07
0.60+
0.56=00.1
0.56=00.4
0,47=001
0.66=0001
FMNIST
7.23:0.56
6.32
9.05
9.13=0.28
5.20
ICFID J
32.18+2111
34.87+5.29
30.53+8.71
37.20=5.50
26.23++7100
36.61+0447
40,00+4.38
23.31+2.77
NML
0,03+0000
0.06+0.00
0.09+0004
0.10:0000
0.014000
0.03+0001
0.29=0002
0,34=0.01
CIFAR-10
FID
81.84:2.27
212.20:14.52
110.51:7.70
61.9::3.99
199.28457.16
34.1341.13
31.5010.75
20.61:0.40
ICFID
139.20+2209
305 3245.05
215.63#11.16
124.27
262.54+59.29
95,43++3.58
81,25+11.55
71,23+66.6
Table 2: Performance comparison on imbalanced attributes
Dataset
Matric
InfoGAN
DeLaGAN
Del.iGAN+
ClusterGAN
SCGAN
CD-GAN
PGMGAN
SLOGAN
0.58=0007
0.65+0.01
0.59+0.01
0.66+00.66
FMNIST-
FID
5.40
7.05::049
6.33:20.44
5.61#0.17
5.01:0020
9.34
11.80:0.43
5.29
ICFID
43.69-1084
36211-3007
35.41
36.94+581
44.48-21.62
39.31+1.18
77.30+8.60
32.46+3.18
NMI
0.05+0.01
0.00=0.00
0.03++0.05
0,22+0002
0.00+0.00
0.22+0.08
0.42++0.03
0.69:0.02
CIRAR-2
FID
51 53
131733400.9
115.19417.95
36 36.62:2.16
45, 15.28
36.40+1.01
19.76
29.00 .9..9
ICFID
88.49+685
18631+-28.11
173.81+18.99
75.52++882
88.58+4.57
76.91+1.07
57.06+3.31
45.83+3.03
aa) (1:2)
(b) CelehA- HQ (1.7:)
Figure 3: Generated high-fidelity images from SLOGAN on (a) AFHQ and (b) CelebA-HQ.
imbalanced attributes. For example, we used two classes of the MNIST dataset (0 vs. 4, referred to as MNIST-2), two classes of the CIFAR- dataset (frogs vs. planes, referred to as CIFAR-2), and five clusters of the FMNIST dataset (rruse)))) (Bag). (T-shir//top, Dress), (Pullover, Coat, Shirt), (Sneaker, Sandal, Ankle Boot), referred to as FMNIST-5 with an imbalance ratio of 1:1:2:3:3). Details of the datasets provided in Appendix
Although SLOGAN and other methods do not utilize labels for training, the data in experimental settings have labels predefined by humans. We consider that each class of dataset contains distinct attribute. Thus, the model performance was measured using classes of datasets. The number of latent components or the dimension of the discrete latent code (K) was set as the number of classes of data.
4.2 EVALUATION METRICS
The performance of our method was evaluated quantitatively in three aspects: (1) whether the model could learn distinct attributes and cluster real data (i.c., cluster assignment), which is evaluated using normalized mutual information (NMI) (Mukherjee al. 2019) (2) whether the overall data distribution p(x,) could be estimated (i.e., unconditional data generation), which is measured using the Fréchet inception distance (FID) (Heusel al., 2017). and, most importantly, (3) whether the data distribution for each attribute p(x,le) (e) could be estimated (i.e., unsupervised conditional generation).
For unsupervised conditional generation, it important to account for intra-cluster diversity as well as the quality of the generated samples. We introduce modified version of FID named intra-cluster Fréchet inception distance (ICFID) described in Algorithm 2 We calculate FIDs between the real data of each class and generated data from each latent code (a mixture component for DeLiGAN and SLOGAN, and category for other methods). We then greedily match a latent code with a class of real data with the smallest FID. We define ICFID the average FID between the matched pairs and use as an evaluation metric for unsupervised conditional generation. ICFID additionally provides class-cluster assignment (i.e., which cluster is the closest the class).
7

---

Published as a conference paper at ICLR 2022
(b)
Table 3: Effectiveness U2C loss
NMI
ICFID
Dataset
Ablation
ICFID
62
SLOGAN w/o EU2C
78.26
CIFAR-10
SL.OGAN
71.23
Ratio of planes (10-x
Ratio of planes (10xx:))
MNIST-2 (7:3)
SLOGAN w/o Eu2c
9.43
SLOGAN
ClusterGAN
CD-GAN
SLOGAN
5.91
Figure Performance comparison with respect to the imbalance ratio on (a) cluster assignment and (b) unsupervised conditional generation.
SLOGAN w/o tU2C
X
Synthetic
SLOGAN
Table 4: Effectiveness of implicit reparameterization
Dataset
Ablation
Ty-0 (groand-trath 0.7)
ICFID
DeLiGAN with Éu20 DeLiGAN with fu2c and implicit update SLOGAN
0.50
60.51
CIFAR-2 (7:3)
1.00
86.48
0.69
45.83
4.3 EVALUATION RESULTS
We compared SLOGAN with InfoGAN (Chen et al. 2016), DeLiGAN Gurumurthy et al. 2017). ClusterGAN Mukkeerjee et all 2019), Self-conditioned GAN (SCGAN) (Liu et al. 2020). CD-GAN
(Pan et al. 2021). and PGMGAN (Armandpour et al. 2021) Following Mukherjee et al. (2019), we used k-means clustering on the encoder outputs of the test data calculate NMI. DeLiGAN has encoder network; hence the pre-activation of the penultimate layer of D was used for the clustering metrics. For fair comparison, we also compared DeLiGAN with an encoder network (referred to as DeLiGAN+). The same network architecture and hyperparameters (e.g., learning rate) were used across all methods for comparison. Details of the experiments and DeLiGAN+ are presented in Appendices and D.5 respectively.
Balanced attributes We compare SLOGAN with existing unsupervised conditional GANs on datasets with balanced attributes. As shown in Table complete version given in Appendix A. SLOGAN outperformed other GANs, and comparisons with methods with categorical priors (ClusterGAN and CD-GAN) verified the advantages of the mixture priors.
Imbalanced attributes In Table 2 (The complete version is presented in Appendix A.2 we compare SLOGAN with existing methods on datasets with imbalanced attributes. ICFIDs of our method are much better than those of other methods, which indicates that SLOGAN was able to robustly capture the minority attributes datasets and can generate data conditioned on the learned attributes. In CIFAR-2 (7:3), the ratio of frog and plane is 7 to 3 and the estimated is (0.6940.00., 0.31+0002), which are very close to the ground-truth (0.7, 0.3). Figure 3 (a) shows the images generated from each latent component of SLOGAN AFHQ (Cat:Dog=1:2). More qualitative results are presented in Appendix A.7
Performance with respect to imbalance ratio We compared the performance SLOGAN with competitive benchmarks (ClusterGAN and CD-GAN) by changing the imbalance ratios of CIFAR-2 from 9:1 to 1:9. SLOGAN showed higher performance than the benchmarks on cluster assignment (Figure 4 (a)) and unsupervised conditional generation (Figure for all imbalance ratios. Furthermore, our method shows a larger gap in ICFID with the benchmarks when the ratio of planes is low. This implies that SLOGAN works robustly in situations in which the attributes of data are highly imbalanced. We conducted additional experiments including interpolation in the latent space, benefits of ICFID. The results of the additional experiments shown in Appendix A
8

---

Published as a conference paper at ICLR 2022
Table 5: Quantitative results of SLOGAN on CelebA
Male (1:1)
Male (1:1)
Eyeglasses (14:1)
Imb. ratio
NMI 1
0.65=0.01
0.29:00.07
FID L
5.1810.20
5.83:10.44
(b) Eyeglasses (14:1)
ICFID
11.0010.66
35.57:5.110
Figure 5: Qualitative results of SLOGAN CelebA.
T Ty=0
0.56++0.02
0.82++0.04
4.4 ABLATION STUDY
U2C loss Table 3 (The complete version is given in Appendix A.33 shows the benefit of U2C loss on several datasets. Low-level features (e.g., color) of the CIFAR dataset differ depending on the class, which enables SLOGAN to function to some extent without U2C loss on CIFAR-10 10. In the MNIST dataset, the colors of the background (black) and object (white) are the same, and only the shape of objects differs depending on the class. U2C loss played an essential role on MNIST (7:3). The modes of the Synthetic dataset (Figure D are placed adjacent to each other, and SLOGAN cannot function on this dataset without U2C loss. From the results, we observed that the effectiveness of U2C loss depends on the properties of the datasets.
Implicit reparameterization To show the advantage of implicit over explicit reparameterization. we implemented DeLiGAN with U2C loss by applying explicit reparameterizatioo on μ and E. Because the mixing coefficient cannot be updated with explicit reparameterization to the best of our knowledge, we also implemented DeLiGAN with U2C loss and implicit reparameterization on p using Equation 7 In Table 4 SLOGAN using implicit reparameterization outperformed explicit reparameterization. When implicit p update was added, the prior collapsed into a single component (Ty== 1) and ICFID increased. The lower variance of implicit reparameterized gradients prevents the prior from collapsing into a single component and improves the performance. Additional ablation studies and discussions are presented in Appendix A.3
4.5 EFFECTS OF PROBE DATA
CelebA ResGAN We demonstrate that SLOGAN can learn the desired attributes using a small amount of probe data. Among multiple attributes which co-exist the CelebA dataset, we chose Male (1:1) and Eyeglasses (14:1). We randomly selected 30 probe images for each latent component. = represents the learned mixing coefficient that correspond the latent component associated with faces without the attribute. As shown in Figure 5and the desired attributes. Additional experiments on attribute manipulation are shown in Appendix A.3
and Table 5) we observed that SLOGAN learned
CelebA-Ho -HQ StyleGAN2 StyleGAN2 (Karras al. 2020) differs from other GANs in that the latent vectors are used for style. Despite this difference, the implicit reparameterization and U2C loss can be applied the input space of the mapping network. On the CelebA-HQ dataset, used 30 male and 30 female faces as probe data. As shown in Figure 3 (b), SLOGAN successfully performed on high-resolusee images and recent architecture, even simultancously with imbalanced attributes.
CONCLUSION
We have proposed method called SLOGAN generate data conditioned on learned attributes on real-world datasets with balanced or imbalanced attributes. We derive implicit reparameterization for the parameters of the latent distribution. We then proposed a GAN framework and unsupervised conditional contrastive loss (U2C loss). We verified that SLOGAN achieved state-of-the-art unsuper- vised conditional generation performance. In addition, a small amount of probe data helps SLOGAN control attributes. In future work, we will consider principled method to learn the number and hierarchy attributes in roal-world data. addition, improving the quality of samples with minority attributes is important avenue for future research unsupervised conditional GANs.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work was supported by the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Scoul National University in 2021, Institute of Information & communications Technology Planning & Evaluation (ITP)) grant funded by the Korea governmen((MSIT) NNO.2021- 0-01343, Artificial Intelligence Graduate School Program (Seoul National University)], and AIR Lab (AI Research Lab) in Hyundai Motor Company through HMC-SNU AI Consortium Fund.
REFERENCES
Mohammadreza Armandpour, Ali Sadeghian, Chunyuan Li, and Mingyuan Zhou. Partition-guided gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5099-5109, 2021.
Georges Bonnet. Transformations des signaux aléatoires travers les systemes non linéaires sans mémoire. Annales des Télécommunications, volume 19, pp. 203-220. Springer, 1964.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In Neural Information Processing Systems (NIPS), 2016.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2020.
Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterization gradients. Advances in Neural Information Processing Systems, 31, 2018.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pp. 5769-5779, 2017.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 166-174, 2017.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR 06), volume 2, pp. 1745-1742. IEEE, 2006.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Proceedings the 31st International Conference on Neural Information Processing Systems, pp. 6629-6640, 2017.
Uiwon Hwang, Dahuin Jung, and Sungroh Yoon. Hexagan: Generative adversarial nets for real world classification. In International Conference on Machine Learning, pp. 2921-2930. PMLR, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztton with International Conference Learning Representutions, 2017.
In Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for
representation learning. In International Conference on Learning Representations, 2021.
Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. In Advances in Neural Information Processing Systems, 2020.
10

---

Published as a conference paper at ICLR 2022
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 2017.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 5110881 19, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11) (1111)):277222222 1998.
Wu Lin, Mohammad Emtiyaz Khan, and Mark Schmidt. Fast and simple natural- natural-gradient variational inference with mixture of exponential-famil approximations. International Conference on Machine Learning, pp. 3992-4002. PMLR, 2019a.
Wu Lin, Mohammad Emtiyaz Khan, and Mark Schmidt. Stein's lemma for the reparameterizzatoo trick with exponential family mixtures. arXiv preprint arXiv:1910.13398, 2019b.
Wu Lin, Mark Schmidt, and Mohammad Emtiyaz Khan. Handling the desitive-defiiite constraint in the Bayesian learning rule. In Proceedings of the 37th International Conference Machine Learning, pp. 61 6116-6126 26, 2020.
Steven Liu, Tongzhou Wang. David Bau, Jun- Zhu, and Antonio Torralba. Diverse image generation via self-conditioned gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14286-14295, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In International
Conference on Learning Representations, 2018
Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan, Clustergan: Latent space clustering in generative adversarial networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4610-4617, 2019.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference machine learning, pp. 2642-2651. PMLR, 2017.
Lili Pan, Peijun Tang, Zhiyong Chen, and Zenglin Xu. Contrastive disentanglement in generative adversarial networks. arXiv preprint arXiv:2103.0366 2021.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180. PMLR, 2019.
Robert Price. A useful theorem for nonlinear devices having gaussian inputs. IRE Transactions on Information Theory, 4(2):69 1958.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Represen- tations, 2016.
Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. In Advances in Neural Information Processing Systems, 2020.
Tingyi Wanyan, Jing Zhang, Ying Ding, Ariful Azad, Zhangyang Wang, and Benjamin Glicksberg. Bootstrapping your own positive sample: Contrastive learning with electronic health record data. arXiv preprint arXiv:2104.0295:
2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 170. 2017.
11

---

Published as a conference paper at ICLR 2022
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations. 2018.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33, 2020.
Huasong Zhong, Chong Chen, Zhongming Jin, and Xian- SannS Hua. Deep robust clustering by contrastive learning. arXiv preprint rrXi::2008.03030, 2020.
12

---

Published as a conference paper at ICLR 2022
A ADDITIONAL RESULTS AND DISCUSSION
We also compared the proposed method with WGAN (Arjovsky et al. (017) with the proposed method. We used the pre-activation of the penultimate layer of D for the clustering metries because WGAN has no encoder network. We could not measure ICFID of WGAN because it cannot perform unsupervised conditional generation. In addition various datasets, we also used the 10x_73k dataset (Zhnng et al.) (017), which consists of RNA transcript counts. From the results of the clustering performances on the 10x 73k dataset, we show that SLOGAN learns useful imbalanced attributes and can be helpful in the use of unlabeled biomedical data.
A.1 PERFORMANCE COMPARISON ON BALANCED ATTRIBUTES
Table 6: Performance comparison on balanced attributes
Dataset
Metric
WGAN
InfoCANN
DeLIGAN
DeLiGAN+
ClusterGiNN
SEGAN
CD-GAN
PGMGAN
SLOCAN
XMI
0.78
0.9011001
0,7010.05
0,772000
0.8-0002
0,74
0.8710003
0.16-0.77
0.92.0.00
MNIST
FID
3.05.0200
1,72
19920.12
2,0010166
1.71=007
3,06
2.751000
5.76-167
167:1.15
ICFID
5.5690.71
25
5.64+0.99
5.12-007
16.6522.1
7.034024
53,40-12.49
4.99:0.19
NMI
0.64::003
0:7+93077
061
0.56:0001
0,564004
0.470001
0.66:8.01
FMNIST
FID
5.74.049
5,28
6.6510045
7,23180.66
6.32-0225
5.07
9:0510.11
9.13-0028
5,2011.33
IFFID
NIA
32.1822.11
34.87.1..99
30,5318.1
37.2015.50
26.23:27.00
36.61
40.00-..78
23.31:1277
NMI
0.14:0002
0.05++003
1315+0013
0,12+0012
0.34-n.22
0.00-00:00
0
0.67-0.00
0.78++0.3
CIFAR-2
FID
29.54:00.99
58.84213..11
338.97470.855
11695419422
36.28:1.12
39,44:1.772
34.454074
29.49:0.51
28.99::0.36
ICFID
NIA
9197:14.21
261.6677.28
153.19417.71
47.02::1.85
71.5415541
43.98:1.47
35.67:0041
35.68
NMI
0.27:0066
0.03
0:06:::000
0.09::0.04
0.10=0.00
0.01:0.00
0.03:000
0.29-0.02
0.3416.01
CIPAR-IO
FID
20.56-0.76
81 21.84-2.23
212.20-4455 53
110.51+7.70
61,97+3499
199.28 19.28-557.16
34.13+1133
31.50-00.8
20.61 +0.40
ICFID
NIA
139. 139.20.22.09
308.32215.05
215.62211.16
124.2715595
262.54159.29
95.43313.58
81.25:11.355
71.23.16.6
A.2 PERFORMANCE COMPARISON ON IMALANCED ATTRIBUTES
Table 7: Performance comparison on imbalanced attributes
Dataset
Metric
WGAN
InfoGAN
DeLEGAN
DeLiGAN+ ClastrGAN
SCGAN
CD-GAN
PGMGAN
SLOGAN
MNIST-2
(7:3)
ICFD
NMI
FID
0.9020.00
4.27+0.19 19
N/A
36 10.35
0.28
4.92+085
25.32
0.90200:4
4.111064
26.11114
0.48:00.1
4.63-2.00
25.41=1.02
0.27:0.19
4.25+1.06
16.47+151
0.6720011
4.34+077
26.1+2247
4.67+1922
0.41:005
890--1422
14.8229.66
0.79=0.1
0.9228.05
4.12+0.86
0.6520000
0
0.65-:0.11
0.60.0002
0.601006
0.5910011
0.24-0022
FMNIST-5
FID
6.55.0..0
5.4010.114
7,0510.49
6.33.00.4
5.6129.17
5.0110.20
9.341056
11,801043
5.29.0. 16
ICRD
N/A
43 43.60 10.54
36.21:3.00
35.1:0099
36.94:5.81
44 4844811000 62
30.3 10.31:11118
77. 77:50=8:0
32.46-3315
10x. 73k
NMI
0.22G0:44
0,4250.06
0.61
0.60=:0.1
0.66
0.4720.02
0.68::000
0.33=0007
0.76:18.02
CIFAR-2
(7:3)
ICHDD
NMI
29.16-0000
0.09
N/A
51.30+251
88.49+685
0,0510.01
131.73550.88
18631-228.11
0,00110000
178.1#18829
0,03:0003
36.6242.16
75.52:4.82
0,22
58.58+4.55 57
0.001:000
7691=1007
0.22:005
1.1
57.06e3.31
0.42:00.5
29.199:6.73
45.83-3303
0.6911.022
CIFAR-2
(9:1)
ICFID
NMI
FID
29,37.10,53
0.0M10004
N/A
135. 10:310:33
60.76:13.97
0.00:10.00
205.2611 26: 10.93
129:50:128.33
0.02210:02
139.75147.15
196 196:00:1:7.6 36
0.09-0.11
41.69.0.83
13331:2..0
0.02:0.01
123, 033.5556.55 56
50.45
0:001:000
128 463
38, 15
0.05::003
101 68::3.87
30.23::1.31
0.16=0.03
29, 29.17
1..75:1.87
0,38114.01
A.3 ABLATION STUDY
Table 8 shows the ablation study on SLOGAN trained with CIFAR-2 (7:3), Ty=0 and Ty=1 represent the mixing coefficients of the latent components that correspond to the frogs and planes, respectively, and the ground-truth of Ty=0 is 0.7.
Factor analysis Rows 1-6 of Table TooleBcompe compare the performance depending on the factors affecting the performance of SLOGAN (4,, Σ. p updates, and (vzc). We confirmed that SLOGAN with all the factors demonstrated the highest performance. Among the parameters of the latent distribution, the μ update leads to the highest performance improvement. The intra-cluster Fréchet inception distance (ICFID) of SLOGAN without update (the 5th row of Table 8) indicates that SLOGAN outperformed existing unsupervised conditional GANs even when assuming a uniform distribution of the attributes.
Loss for P update We do not use U2C loss luzc to learn the mixing coefficient parameters p. We construct U2C loss to approximate the negative mutual information -T(C;x,) that can be
13

---

Published as a conference paper at ICLR 2022
Table 8: Ablation study on CIFAR-2 (7:3)
Ablation
yy (ground-trth:
ICFID +
Factor analysis
SLOGAN without Σ. updates, &v20
0.50
84.44
SLOGAN without μs, Σ. p updates
0.50
77.32
SLOGAN without μa update
0.52
73.79
SLOGAN without p update
0.50
63.09
SLOGAN without Σ update
0.69
48.34
SLOGAN without EU2C
0.66
48.82
Implicit reparameterization
DeLiGAN with fu2C
0.50
60.51
DeLiGAN with EU2C and implicit p update
1.00
86.48
Loss for p update
SLOGAN with Éu2C for p update
0.62
52.67
SimCLR analysis
SLOGAN without SimCLR
0.66
49.25
SLOGAN without SimCLR on real data only
0.67
48.41
SLOGAN without SimCLR on both real and fake data
0.69
47.93
Attribute manipulation
SLOGAN with probe data
0.71
44.97
SLOGAN with probe data and mixup
0.70
44.26
SLOGAN
0.69
45.83
Table 9: Effectiveness of U2C loss
Dataset
Ablation
NMI
FID |
ICFID
MNIST-2 (7:3)
SLOGAN w/o Eu2C
0.25
4.62
9.43
SLOGAN
0.92
4.02
5.91
SLOGAN w/o E122 SL.GGAN
0.14
5.27
43.15
FMNIST-5
0.66
5.29
32,46
SLOGAN w/o E12C
0.01
29.18
41.72
CIFAR-2
SLOGAN
0.78
28.99
35.68
CIFAR-2 (7:3)
SLOGAN w/o E22
0.08
30.34
48,82
SLOGAN
0.69
29.09
45.83
SLOGAN w/o E12C SLOGAN
0.08
20.91
78.26
CIFAR-10
0.34
20.61
71.23
decomposed into entropy and conditional entropy as follows:
(z2c(Z) = - -(C = H(C)x)) (C)
(12)
The conditional entropy term reduces the uncertainty of the component from which the generated data obtained. The entropy term promotes that component IDs are uniformly distributed. In terms of p update, the entropy term (CC)) drives p(C)) toward a discrete uniform distribution. Therefore, using εu2c for learning P pulls to discrete uniform distribution and can hinder the learned л from accurately estimating the imbalance ratio inherent the data. the 9th row of Table & we observed that the unsupervised conditional generation performance undermined and the estimated imbalance ratio (my=o) was learned closer to a discrete uniform distribution (0.5) when Ev2O was used for p update.
SimCLR analysis For the colored image datasets, SLOGAN uses the SimCLR loss for the en- coder with only fake (generated) data to further enhance the unsupervised conditional generation performance. The 10th to 12th of Table show several ablation studies that analyzed the effect of SimCLR loss on SLOGAN. SLOGAN without SimCLR still showed at least approximately 35% performance improvement compared to the existing unsupervised conditional GANs (ICFID of
14

---

Published as a conference paper at ICLR 2022
ClusterGAN: 75.52, CD-GAN: 76,91 in Table 2) even considering the fair computational cost and memory consumption. The SimCLR loss shows the highest performance improvement especially when applied only to fake data. SimCLR improved the performance by 7%.
Attribute manipulation As shown in the 13th and 14th rows of Table 8 probe data significantly improved the performance of SLOGAN component. We also confirmed that the mixup applied to the probe data further enhanced the overall performance of model. Figure (a) shows the data generated from SLOGAN trained CIFAR-2 (9:1) without the probe data. With extremely imbalanced attributes, SLOGAN mapped frog images with white background onto the same component as airplanes in its latent space, When we use 10 probe data for each latent component, as shown in Figure (b). frogs with a white background were generated from the same latent component as the other frog images.
CIFAR-2 (7:3) with 10 probe data for each latent
(a) SLOGAN (without probe data)
(b) SLOGAN with probe data & mixup
Figure 6: Effects of attribute manipulation on unsupervised conditional generation. Left and right images visualize generated images from different latent components. The red boxes indicate generated frog images with a white background.
We introduced the feature scale described in Appendix D.I to reinforce the discriminative power of U2C loss. For the MNIST-2 (7:3) dataset, was set to 4. Such a parameter configuration is justified by greedy search in [0.5, 1, 2, 4, 8). The performances of SLOGAN on MNIST-2 (7:3) with different feature scales are shown in Table
Feature scale
Table 10: Ablation study on feature scale
0.5
ICFID 1 14.18
17.03
6.65
33.98
Intuitively, increasing the feature scale & makes the samples generated from the same component closer to each other in the embedding space. From these results, we observed that the optimal choice of the temperature factor enhances the discriminative power of U2C loss.
A.4 STATISTICAL SIGNIFICANCE
We completed the statistical tests between SLOGAN and other methods in Tables and 2 Since the results of the experiment could not satisfy normality and homogeneity of variance, we used the Wilcoxon rank sum test (Wilcoxon| 1992). When p-value 0.05, measured the effect size using Cohen's 's d (Cohen] 2013). We validated that all the experiments are statistically significant or showed large or medium effect sizes, with the exception of FID of InfoGAN vs. SLOGAN for FMNIST in Table I NMI of DeLiGAN+ vs. SLOGAN for FMNIST-5 in Table 2
A.5 INTERPOLATION IN LATENT SPACE
We also qualitatively show that the continuous nature of the prior distribution of SLOGAN makes superbly smooth interpolation possible in the latent space. In Figure we visualize images generated from latent vectors obtained via interpolation among the mean vectors the trained latent components. The generated images gradually changed to 3, 5, and for the MNIST dataset, and t-shirt/top, pullover, and dress for the Fashion- -NNIST dataset. In particular, we confirmed that the face images generated from the model trained with the CelebA data changed smoothly. The continuous attributes of real- world data are well mapped to the continuous latent space assumed by us, unlike most other methods using separated latent spaces induced via discrete latent codes.
15

---

Published as a conference paper at ICLR 2022
3
3
3
3 5
5
5
5
5
5
5
5
S
5
5
8
8
8
8
Figure 7: Interpolation in the latent space of the proposed method. For the MNIST and Fashion- MNIST datasets, we selected three mean vectors in the latent space and generated images from linearly interpolated latent vectors. For the CelebA dataset, we used 30 probe data and mixup for each latent component with attributes such as Black hair (3:1) and Male (1:1).
A.6 BENEFITS OF ICFID
DeLiGAN and ClusterGAN trained on the MNIST-2 (7:3) exhibited comparable FIDs to SLOGAN
(DeLiGAN: 4.21, ClusterGAN: 4.25, and SLOGAN: 4.02); however they showed ICFIDs approxi-
mately four times higher (DeLiGAN: 25.34, ClusterGAN: 25,61, and SLOGAN: 5.91). From the data generated from each latent component of DeLiGAN and ClusterGAN Figure X (b) and (c), we
confirm that the attributes were not learned well in the latent space of DeLiGAN. By contrast, from the data generated from each latent component SLOGAN presented in Figure S (a), SLOGAN successfully learned the attributes in its latent space. This shows that ICFID is useful for evaluating the performance of unsupervised conditional generation, In addition, ICFID can evaluate the diversity of images generated from discrete latent code or mode because ICFID is based on FID. As shown in Figure 9 when a mode collapse occurs, the diversity of samples decreases drastically, and DeLiGAN trained on the CIFAR-2 (7:3) shows approximately twice the ICFID than those of InfoGAN and ClusterGAN (DeLiGAN: 186.31, InfoGAN: 88.49, and ClusterGAN: 75:52).
0
4
4
4
0
0
4
4
6
0
0 4. 4 4 G 4 4 # 0 d O 2 4 4 0 A 0
4
400
0
4
4
4
0 0 0
4
0
0 0 0 0 D 0 0
0
0
144
0
4
4 4 1
4
4 0 4 4 4 M 4 O 4
O
0
4
4
0 0 0 0 0 0
0
0
4
H
44
4
4
4/4
4
0 0
0
4 4: 4
0 0
4
(a) SLOGAN
(b) DeLiGAN
(c) ClusterGAN
Figure 8: An example where ICFID is useful. The left and right images show generated images from each latent code of (a) SLOGAN, (b) DeLiGAN and (c) ClusterGAN trained on the MNIST-2 (7:3) dataset, respectively.
DeLiGAN
Figure 9: Another example where ICFID is useful. The left and right images show generated images from each latent component of DeLiGAN trained on the CIFAR-2 (7:3) dataset.
16

---

Published as a conference paper at ICLR 2022
A.7 QUALITATIVE RESULTS
Synthetic data Figures 10 and 1| shows the synthetic data and data generated by SLOGAN. Each color denotes generated data from each mixture component of the latent distribution. As the training progresses, each mixture component in the latent space is more strongly associated with each Gaussian distribution in the data space.
Generated images and latent spaces Figures 12
18 19 the left plot of Figure 15 and
the upper plot of Figure 16 show the images generated from each latent component of SLOGAN trained on various datasets. Figure 14 the right plot of Figure 15 and the lower plot of Figure 16 visualize 1,000 latent vectors of SLOGAN trained on the MNIST, Fashion-MNIST, MNIST-2 (7:3), and FMNIST-5 datasets using 3D principal component analysis (PCA). Each color represents the component with the highest responsibility, and each image shows the generated image from the latent vector. As shown in Figure 14 similar attributes (e.g., 4, 7, and 9) are mapped nearby components in the latent space.
Comparisons with the most recent methods We compare our method with the most recent methods such as CD-GAN (Pan et al.] 2021) and PGMGAN (Armandpour et al. (222I) on the CIFAR-2 (7:3) dataset. From the results shown in Figure 20 we qualitatively confirm that SLOGAN learns imbalanced attributes of the dataset most robustly.
Highly imbalanced multi-class data We trained our method on highly imbalanced multi-class datasets by setting class 8 of the MNIST dataset to very low proportions of the other nine classes (e.g., 10:10:10:: 10:10:10:1:: 10: 1:1110100:1110 and 10:1:::::0:1::::::110::::::::::: 10: 10:2:10). When class 8 is 0.1 fraction of the other nine classes, images of class 7 with horizontal line outnumber images of class 8, and SLOGAN identifies with horizontal line as more salient attribute than 8 as shown in the red boxes in Figure 21 (a). On the other hand, when class 8 is 0.2 fraction of the other nine classes, images of class 8 outnumber images of class 7 with a horizontal line. Therefore, SLOGAN successfully identifies 8 salient attribute as shown in the red box in Figure (b).
Qualitative analysis with various imbalance ratios Figure 22 shows generated images from each latent component of SLOGAN trained on the AFHQ dataset. For various imbalance ratios of cats and dogs, we qualitatively analyze SLOGAN without using probe data. When the imbalance ratios are 1:1 and 1:2, SLOGAN identifies cat/dog as the most salient attribute and learned the attribute successfully presented in Figure 22 (a) and (b). When the imbalance ratio is 1:5, SLOGAN discovers folded ears as the most salient attribute as shown in Figure (c).
Training data
Step
Step 1000
Slep 5000
Step 10000
Figure 10: Synthetic dataset and samples generated by SL.OGAN at 0, 1000, 5000, and 10000 steps.
@a) = 0.01
(b) = 0.05
(c) 0.10
Figure II: Generated samples from each latent component of SLOGAN trained on the synthetic datasets with variances (a) 0.01I, (b) 0.05/, and (c) 0.10/.
17

---

7 Published as a conference paper at ICLR 2022
3 3 333333
2
9
9
3 3 3
2
2
2
2
2
9
9
9 9
3 333
3
6
2
2
7
9
9
9
333
6
2
9
9
3 3333334
2
2
2
9
9
3 333333 3333
2 2 2
2-
2
9
9
3 3
333
199999999
2.
2
2
8
2
9
O
3 3 3 3 3338
2.
2
2
2
2
O
5 5 555555
8 8888888
7
7 7 7 7 777
7 7 7 7
F
5 5 5 S 5 S
8 8
8 8 &
7
5 5 $
5 5
8
8
888888888
7
7
7 7
5 5 5
5 5 5
8
8
8
2
7
5 5 055555
8
8
8
8
8
8
7
7
7
H
5 5 5 555555
8
8
8
8
8
7
7
7
7
7 7
7 7
5
5 5 5 $
8
8
8
8
8
7
7
7
7
5 5
Figure 12: Generated images from each latent component of SLOGAN trained the MNIST dataset.
Figure 13: Generated images from each latent component SLOGAN trained on the Aishion-MNIST dataset.
MNIST
FMNIST
Figure 14: 3D PCA of the latent spaces of SLOGAN trained on the MNIST and FaShion-MNIST datasets.
18

---

Published as a conference paper at ICLR 2022
4
4
Y
4
4
4
4
4
4
4
4
Y
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4 4
4
4
4
4
4
4
4
4
Figure 15: Generated images from each latent component and 3D PCA of the latent spaces of SLOGAN trained on the MNIST-2 (7:3) dataset.
Figure 16: Generated images from each latent component and 3D PCA of the latent space of SLOGAN trained on the FMNIST-5 dataset.
C-FAR-2 (7:3)
C-2 (9:1)
Figure 17: Generated images from each latent component of SLOGAN trained on the CIFAR-2 (7:3)
and CIFAR-2 (9:1) datasets.
19

---

Published as a conference paper at ICLR 2022
(a) Male (1:1)
(b) Eyeglasses (14:1)
Figure 18: Generated images from each latent component of SLOGAN trained the CelebA dataset. We used 30 probe data ((a) Female vs. Male, or (b) Faces without eyeglasses vs. Faces with eyeglasses) and mixup for each component.
20

---

Published as a conference paper at ICLR 2022
T
2
Figure 19: Generated images from each latent component of SLOGAN trained the CelebA-HQ (256x256 dataset. We used probe data (Female vs. Malc) and mixup for each component.
21

---

Published as a conference paper at ICLR 2022
(a) CD-GAN
(b) PGMGAN
(c) SLOGAN
Figure 20: Generated images from the most recent methods including (a) CD-GAN, (b) PGMGAN,
and (c) SLOGAN trained the CIFAR-2 (7:3) dataset.
0
A
2
> 2
2
3
3
4
4
4
4
/
2
2
R
R
2 2
2
3
3
3
4
4
D
1
2
2
2
2
2 2
2
3
3
4
4
4
0 8
2 2
2 2
2
3
3
4
4
D
2
3
3
4
4/
2
2 2
2
2
2
4
/
4
0
2
2 2
2
2 2
3
4
4
4 4 9
2
2
3
5
55
6
7
7 7
7 7 7
7 7
7
+
7
9
7
9 9
9
9
9
5 5 S 55555
6
4
7
7
7
7
7
2
7
7 4
4
9
9
9
9
9
5 1555555 5 033555555
55555
6 6
7
7
7
7
7
4
9
9
9
5 5 199955
9 999996660
1777 7
77
7
7
4
7
9
9
9
9
5 135555555
9
7
7 7
7
7
7
9
9
9
9
9
9999999
7 9999
7
7
7
9
9
9
9
5555555 1555555
7
7 7
7
7
7
4
9
9
5555555
555555
7
9
a Class is 0.1 fraction of the other nine classes
2
2
2
3
a
2 2
3
3
3
3
3
4 4
/
2
2 2
2
2
3
3
5
3
3
3
4
4
4
/
2
2
2
2 2
2
2
3 3033330
3
3
4
4
4
2
2
222
3 3
3
4
4
4
4
2
2
2
2 2
2 A
2.2
3
3
3
8
4
4
4
4
40
2
2
2 2 2 2
3
3 3
4
/
4
4
2
2
a
2
3
3
8
3
4
4
4
Y
4 4/
4
2 2
2
2 7
2
3 3
3
3 3
4
//
4
4
6
6
6
6
6
7
7 7
7 7
3
9
9
9
9
9
9
9
9
5 5 55
6
6
4
7
7
7
8
8
4
9
9
9
9
9
5 5 1555 5555555
9
7
2
9
9
9
9
9
5 5555555540
5 5 555555550 5555555
99 7999999
3
7
9
9
9
9
9
9
999999
8
8 8
4
4
4
9
9
9
5 5 55555
6 99999999
9 99
7 7
7
8
8
9
9
9
9
9
9
5 5 155555555 55555555
6
7 7
7
7
8
8
9
9
9
5555555555 555555555 5
6
6
9
(b) Class 8 is 0.2 fraction of the other nine classes
Figure 21: Generated images from each latent component of SLOGAN trained on the MNIST dataset where class 8 very low fraction of the other nine classes.
22

---

Published as a conference paper at ICLR 2022
(a) Cat: Dog=1:1
(b) Cat:Dog=11:
(c) Cat:Dog= 1 1:5
Figure 22: Generated images from each latent component of SLOGAN trained on Cats and Dogs of the AFHQ (256x256) dataset with various imbalance ratios.
23

---

Published as a conference paper at ICLR 2022
B ADDITIONAL BACKGROUND
Our work is closely related to Stein's lemma and the reparameterization trick for Gaussian mixture and also related to several topics of GAN studies such as representation learning, supervisedduussuprriiid conditional generation, and Gaussian mixture prior.
B.1 SUPERVISED CONDITIONAL GENERATION
Conditional GANs including ACGAN (Odena et al. 2017), projection discriminator (Miyato & Koyama 2018). and ContraGAN (Kang & Park 2020 have led to state-of-the-ar performances in conditional image generation. However, these conditional GANs are trained with supervision and require a large amount of labeled data.
B.2 UNSUPERVISED CONDITIONAL GENERATION
InfoGAN (Chen et al. 2016) introduces latent codes composed of categorical and continuous variables and derives the lower bound mutual information between the latent codes and representations. By maximizing the lower bound, InfoGAN learns disentangled latent variables. ClusterGAN (Mukherjee) jtt al. 2019) assumes a discrete-continuous Gaussian prior wherein discrete variables are defined as one-hot vector and continuous variables are sampled from a Gaussian distribution. The latent distribution clustered in an unsupervised manner through reconstruction loses for discrete and continuous variables. CD-GAN (Pan et al. 22021) is similar to ClusterGAN, but it uses contrastive loss to disentangle attributes. These methods assume uniform distribution of the attributes, and if the imbalance ratio is unknown, these models cannot thoroughly learn imbalanced attributes.
Recently, unsupervised conditional GANs which do not assume uniform distribution of the attributes have been proposed. Self-conditioned GAN (Liu et al. 2020 performs unsupervised conditional generation by using clustering of discriminator features as labels. However, it has no loss to facilitate clustering of discriminator features, and the performance seems be largely influenced by the architecture of the discriminator. PGMGAN (Armandpour et al. (021) applies a contrastive clustering method named SCAN (Van Gansbeke al. 2020) to perform unsupervised conditional generation. The pretrained space partitioner and a penalization loss function are used to encourage the generator to generate images with salient attributes. To avoid degenerated clusters where one partition contains most of the data, PGMGAN uses an entropy term that regularizes the average cluster probability vector to uniform distribution. However, when learning PGMGAN on datasets with imbalanced attributes, it is difficult to adjust the coefficient of the regularizer because the performance seems to be sensitive to the strength the regularizer in practice (c.g., the results on CIFAR 2 (7:3) and (9:1) in Table 7). In addition, clustering results does not seem to be reliable when the space partitioner is trained on datasets where transformations are not available (e.g., the result on 10x 73k in Table
NEMGAN Mishra et al. 2020) argues that considers the imbalance problem of attributes. However, assumes that some labeled samples are provided, which is unrealistic in real-world scenarios. In addition, labeled samples should have the same imbalance ratio as the training data estimate the imbalance ratio. If these samples are not given, NEMGAN is the same method as ClusterGAN. On the other hand, SLOGAN does not require labeled samples learn imbalanced attributes. Even when a small amount of probe data is used to learn specific attributes, imbalanced attributes can be learned with balanced probe data (c.g., 30 male and 30 female faces on the CelebA-HQ dataset).
B.3 REPRESENTATION LEARNING
Representation learning refers to the discovery of meaningful semantics in datasets (Bengio et al. 2013). Perceptually salient and semantically mcaningful representations can induce better perfor- mance in downstream tasks. In particular, Representation learning in generative models enables the learned latent representations to be semantically meaningful without labels in the training data. BiGAN (Donahue et al.) 2017), ALI (Dumoulin al. 2017). and their variants (Jaiswal al. 2018 Belghazi et al.) 2018) are similar to our study in that they add an encoder network to the original GAN framework. The additional encoder may serve as inversion mapping of the generator, and leamedd feature representation is useful for supervised tasks. However, the learned generator cannot perform conditional generation without supervision.
24

---

Published as a conference paper at ICLR 2022
C PROOFS
C.I GRADIENT IDENTITY FOR Pe
Theorem Given an expected loss of the generator L and a loss function for sample E() Rd R. we assume & is continuously differentiable. Then, the following identity holds:
p. - = - (o
(13)
Proof. To prove Theorem D the following lemma (Bonnet's theorem) is introduced.
Lemma Let h(z)) TRd - R be continuously differentiable. q(z) is a multivariate Gaussian
distribution (z; μa, Σ). Then, the following identity holds:
yn [h(z)] = Eg(()) V [z)(z))
(14)
The proof of Lemma? 2 is described by Theorem of Lin et al. (201)). Using Lemma
we show that
K
V u. = Vm, By(x) [((z)] = Vn. Σ (000 [((x)]
(15)
=(ccV Me Bycelc) [[(z)] - p(c) p(c) By(elc) [[((z)]
(16)
(17)
/ ((2)) ((ze))(()) V Vzl(z)dz
(18)
((2)
(()) Vz((z)d2
(19)
Fa(z) (8 Vo((z))
(20)
C.2 FIRST-ORDER GRADIENT IDENTITY FOR E.
Theorem 2. With the same assumptions from Theorem 7 we have the following gradient identity:
V C マ E.g (((2)TT -z- (z- V {())]
(21)
Proof. In order to prove Theorem we introduce the following lemma (Price's 's theorem).
Lemma 3. Let h(z): Rd + and its derivative Vh(z) be continuously differentiable. We further assume that E [h(z)) is well-defined Then, the following identity holds:
Vz [h(z)] = (x)))) Σ Σ 11) 4 h())
Bo(x)) [22(2)]
(22)
The proof of Lemma is presented in Theorem 4 of Lin et al. (011 The rest of the proof is similar with the proof of Theorem Using the first-order gradient identity of Lemma we get
V3,C Vx, Ex(x) [((z)] = Vz. Σ E [[(z)]
(23)
vlc) Byllee) [[(2]]
2 p( Eg(lle)
(1o==uo)VV ((.)
(24)
2 / ((zlc)p(c) (2 me) V {(z)dz
1
(25)
1
/ ((z) ((2c)/p())
No (z - mc) V lzzddz
(26)
2
g(2)
25

---

Published as a conference paper at ICLR 2022
1 (22 (c)) Vo ((z)dz
2 ((2)0(2)0002)
(27)
2 Bule) (ó(z) (o Mc) Ho) [())
(28)
C.3 ENSURING DOSITIVE-DEFINTTEN OF Σ
Theorem 3.
The updated covariance matrix E. = Ec+rAE with the modified update rule specified in Equation 0 is positive-definie if E. is positive- -eeirive-eefiite..
Proof. Because Ee is symmetric and ppsitive-definit, we can decompose Eg = LL" using Cholesky decomposition, where the lower triangular matrix. Then, we can prove the positive-defniteness of the updated covariance matrix as follows:
Σx = E. + 7
(29)
= E. + 7 + AAENE-AAE
(30)
+ +
2 AE A
(31)
E + AL 1 (L" +TL=)
(32)
2
(33)
Let us define U I= 7I--1A
Then, we have the following:
c EJ U²
(34)
where both Eg and U are positive-defnite, concluding the proof.
C.4 GRADIENT IDENTITY FOR Pe
Theorem 4. Let Pe be a mixing coefficient parameter, and the following gradient identity holds:
C= 1Eg |Te (8(z). 1) {(z)]
(35)
Proof. The gradient of the latent distribution with respect to the mixing coefficient parameter is derived as follows:
K
V pp)(Z)) =V V Pc
softmax To [((lc))
Σ
(z)())
(36)
1=1
1=1
where softmax(-) denotes the softmax function (e.g., n, softmax(p:)). Using the above equation, we have
Vp. C = (e [[(z)]] = / (()
(37)
K (1](2))) e(z)dz
f Te
((2lc)
(38)
1=1
= -f Tz((2)
(((lc)
K
g(2l))
((2)
T
((x)
{(2)dz
(39)
1=1
K
=f a(Z))Te
(8(2)c
TzŌ(Z);
Elz)dzz
(40)
==1
26

---

Published as a conference paper at ICLR 2022
Here, Σ/=17, n,,(z), Ex=
p(olaa)) ((x) 1. Plugging this back into Equation 40 obtain
P C / g(z)=Te Te (5(z)c ((z)dd
(41)
Bg(x) (TT (Ó(z); - 1) {(z))
(42)
C.5 SECOND-ORDER GRADIENT IDENTITY FOR Σ.
We consider a generator o OG()) og. discriminator ( D with the piecewise- linear activation functions (e.g. ReLU or LeakyReLUU except the activation function of the output layer of the generator which is the hyperbolic tangent function ((x) = (e" "ee +e=*), and Wasserstein adversarial loss function for each latent vector f(z) = -((o(((()) = l. We note that the following holds except set of zero Lebesgue measure because the piecewise-linear activation functions are linear except where they switch:
0DD doc²
OG Oz
O,
(43)
We provide the second-order gradient identity of the Wasserstein GAN loss for the covariance matrix E. that does not compute second-orde derivatives. However, it is impractical and not included in our method because of the excessive computational cost of Jacobian matrices.
Theorem 5. Given the Wasserstein GAN loss for the generator L and a loss function for sample ((z) Rd + RR we assume f and its derivative (z(z) are continuously differentiable. We further assume that [f(z)] well-defined. Then, the following identity holds:
Vr Vr. By(a) [((z)] -By(s) [T))( V G(z) diag D ac)) VaG(z))
where @ denotes element-wise multiplication of vectors and x denotes the i-th Hadamard (element- wise) power of a vector x.
Proof. From the second-order identity of Lemma we have the following:
Vx,C = (y(t)) [e(z)] Vx. p(i) (2)))) [f(z)]
(44)
Vr, x. Bytzlo) [8(2)] =
2 p(c) (c( [2((2z)
(45)
(g(lc)p(() {Z)
2 Egtx)
(46)
((z)
Ey(z) (x=0(Z)) V2 [z)]
(47)
The Hessian of the sample loss with respect to given as follows:
vel(z) = V, ( = v,
(48)
D
(49)
aG1
0G1
Uz
0D
(o()1
Hoaaa (d')da)
aD
(50)
Oz
0(oc)1
8Gdz
0G4z
881
asa.
Σ iml
8D
(oc)ss
aG.
Σ am
aD
(o'c)s
aG:
(51)
Oz
Dz
Ozd.
8
d.
8D
(o'a):
80,
à
d,
8D Teach
(d'a): 8G,
Øz1
Bool
0=
Uzd.
(52)
8D
( G
)
aG.
8
an
(o'c)
aG.
azd.
#=1
Boc)
dea.
-==
Toa...
27

---

Published as a conference paper at ICLR 2022
where 0'6- V.oG. To simplify an element (V2(())){ -0e for arbitrary j.k {I,... ] we have
0. (Ei= Tlaah: (dc) 8G
BD
DD
DD
de 0D M(aa)l
(53)
Ō2,0(oc)
a(oa). dzj
MoG
a(Ga)i dzj
0D
8Gi (da)l 0z,
Σ
(54)
a(oc) o(oc))
@DD
0G Ozj
dz
aD
M(da)i aG,
(55)
H(c))
Mac)) o(ac); dz₁
8D
PG
(oc))
(56)
M(ac))
Noc))d2j
1=1
aD
8G:
0D
8
(-2()))) dzj doa))
(da):
(57)
O(ac);
8zj (oc))
aD
0G:
0D
((a)); 8G;
(-2(ac))) 0zj M(aa).
(0b))
(58)
O(oc).
(otTe dz,
(59)
d. aD M(d'): aG:
0D M(o'): 0G; 8G;
(60)
Hog) dz, dzk
OoG)), 0G, Ozy Ozk
d2
OD
0G, 8G;
(2( 2(oa)))
(61)
)oa)).
0zj 0zk
d. aD (dc)s 8 =0 0Z,02%
(62)
O(aa);
Therefore, the simplified element is
(V)((2))s = d,
aD (2(06) 2(
8G, 8G;
(63)
Moc).
Ozj O2k
We now vectorize the expression of (Z) follows:
(GG)
aD doG
V2((z) -2
diag
diag (o c3 -oG)
(64)
8z
Oz
Plugging this to Equation 47 the following is obtained:
Vx Bx(x) [T.6(Z). V G diag Do (o σc)) [zG]]
(65)
28

---

Published as a conference paper at ICLR 2022
D METHODOLOGICAL DETAILS
D.1 ADDITIVE ANGULAR MARGIN
To enhance the discriminative power of U2C loss, we adopted the additive angular margin (Deng
et al. 2019 as follows:
exp(s cos cos(O.; + m))
(v2c(2') = llog (exp(s cos(0,; m]) + jay exp(s cos cos 01;)}
(66)
where denotes the feature scale, and is the angular margin. The feature scale and the coefficient
of U2C loss À are linearly decayed to 0 during training, so that SLOGAN can focus more on the adversarial loss training progresses.
D.2 CLUSTER ASSIGNMENT
In Section 3.3, we chose cos 0se the critic function assuming that it is proportional log p(c p(c)xg). the real data distribution p(x,) and the generator distribution p(x,) ) are sufficiently similar via adversarial learning, the cosine similarity between Exx,) and Me also be considered proportional to log p(c(x,). Therefore, for real data, we obtain the probability for each cluster as follows:
exp(cos 0z)
p(cx,) = Σ exp(cos 0&)
67
where cos 02 = E(x,) Exx
(xl,)ll p ll is the cosine similarity between E(x,) and pa. The data can then be assigned to the cluster with the highest probability (i.e., argmax, ((c]x,)).
D.3 ATTRIBUTE MANIPULATION
We utilized mixup Zhang et al. 2018 to make the best use of a small amount of probe data when manipulating attributes. Algorithm describes the procedure for using mixup for attribute manipulation when K = 2. We applied the same feature scale and angular margin to Lm, as shown Equation 66
The number of iterations for the mixup (T') was set to five.
Algorithm 3 Attribute manipulation
Initialize probe data with the desired attribute X f- Jim and X f- {x-=1)== LM Initialize probe data without the desired attribute X X {x==0]= and X t {x-ooii for each mixup iteration in {1, .... 7} do
X 11 MIXUP X - PERMUTE X
X 0 U MIXUP = PERMUTE(X
end for
for each augmented data index j in {1, 1..... M(T 1)) do
cos 0% E((x Mo/l [x²) [0x==)0)1
cos 0g E(x pi (x
cos 8i - F(x((1)) po/ (x) 1)ll
cos i1 + E(x(_1)
(xe)
end for
Lm M(T+1) ENT
M(T+1) log exp(s-cos((),,
exp(s +m)) txx((= +m)
texp(ss texp(s-cos 0(0) exp(s-cosl (f +m)
Minimize L with respect to F, G. and
D.4 SIMCLR
For the CIFAR and CelehA datasets, we used the SimCLR loss (Chen al. 2020) for the encoder. We applied color, translation, and cutout transformations to the generated data using DiffAAgment
https://gi commii...oooommmmmmmo//.oooom.. hftt:///tttttaaa/aaaaaaaaafaeiee
29

---

Published as a conference paper at ICLR 2022
(Zhao et al. 2020). The SimCLR loss is calculated using the generated data x/ and augmentation A as tollows:
exp(E(x)) ((A(x))) |
(SiCCLR(Z') log E exp( (x FB (A (x2)) || [|))
(68)
where x G(zl). +Alu2c(z')).
The encoder E' trained minimize # Σ B (Cadv (SimCLR(Z))
D.5 DELIGAN+
Among the existing unsupervised conditional GANs, DeLiGAN lacks an encoder network. Therefore, for a fair comparison, we added an encoder network and named it DeLiGAN+. We set the output dimension of the encoder to equal the number of mixture components the latent distribution. For the i-th example in the batch, when the c--th mixture component of the latent distribution is selected, DeLiGAN+ learned through the following objective:
B - c.)) - AOE cf log E(G(z', c.)]]
min max D B
(69)
G. G.E.Me
where c, is the one-hot vector corresponding to o and AGE is the coefficient of the cross entropy loss. We set ÄOE to 10 in the experiments.
D.6 EVALUATION METRIC
Cluster assignment We do not use clustering purity which is an evaluation metric for cluster assignment. To compute the clustering purity, the most frequent class in the cluster is obtained, and the ratio of the data points belonging to the class is calculated. However, if the attributes in the data are imbalanced, multiple clusters can be assigned to a single class in duplicate, and this high clustering purity misleads the results. Therefore, we utilized the normalized mutual information (NMI) implemented scikit-lear
Unconditional generation FID has the advantage of considering only sample quality but also diversity, whereas Inception score (IS) cannot assess the diversity properly because does not compare generated samples with real samples (Shmelkov al. 2018). Therefore, we used FID the evaluation metric for unsupervised generation.
Unsupervised conditional generation If attributes in data are severely imbalanced, FID does not increase (deteriorate) considerably even the model does not generate data containing the minority attributes. Therefore, the FID cannot accurately measure the unsupervised conditional generation performance for data with severely imbalanced attributes. We introduce ICFID to evaluate the performance of unsupervised conditional generation. When calculating ICFID, multiple clusters cannot be assigned to single class in duplicate. Therefore, if data of single class generated from multiple discrete latent variables or modes, the model shows high (bad) ICFID.
hetrrcee/chsseer/__ppevvvee sed.yy
30

---

Published as a conference paper at ICLR 2022
E IMPLEMENTATION DETAILS
E. GENERAL SETTINGS AND ENVIRONMENTS
For simplicity, denote the learning rate of G 1 and the learning rate of Σ as y. Throughout the experiments, we set the learning rate of E. to and D to 4nj using the two-timescale update rule (TTUR) (Heusel et al. (017). We set the learning rate of μ 10%, and the learning rate of p to y. We set B to 64 and the number of training steps to 100k. To stabilize discriminator learning, we used Lipschitz penalty (Petzka et al. 2018) for the synthetic, MNIST, FMNIST, and 10x_73k datasets, and adversarial Lipschitz regularization (Terjek] 2020) for the CIFAR-10 and CelebA datasets. We repeated each experiment 3 times and reported the means and standard deviations of model performances. Hyperparameters are determined by grid search. We used the Adam optimizer (Kingma & Ba 2014) for training D, G, and F,, and gradient descent optimizer for training μ., Σ, and p. The experiments herein were conducted on machine equipped with an Intel Xcon Gold 6242 CPU and an NVIDIA Quadro RTX 8000 GPU. The code is implemented in Python 3.7 and Tensorflow 1.14 (Abadi et al. 2016).
E.2 SYNTHETIC DATASET
For the synthetic dataset, we first set the mean of eight 2-dimensional Gaussian distributions as (0, 2).
(V2, 2). (2, 0), (v2, 2V (0, 22). (-v2, - 22),, (-2, 0). and (-v2, 22), and the variance as 0.01I. In Figure II we also the variances as 0.05/ and 0.1I. The number of data sampled from the Gaussian distributions was set to 5,000, 5,000, 5,000, 5,000, 15,000, 15,000, 15,000, and 15,000. We scaled total of 80,000 data points to a range between -1 and 1. Table [T shows the network architectures of SLOGAN used for the synthetic dataset. Linear denotes a fully- conneced layer with n output units. BN and SN denote batch normalization and spectral normalization, respectively. LReLU denotes the leaky ReLU. We set À = 4, 0.001, 7 0.01, 2, and m 0.5.
Table 11: SLOGAN architecture used for the synthetic dataset
G
D
E
Z Rd4
E R²
E R2
Linear 128 BN ReLU
Linear 128 LReLU
Linear 128 SN LReLU
Linear 128 + BN ReLU
Linear 128 LReLU
Linear 128 SN + LReLU
Linear 2 Tanh
Linear
Linear 64 +
E.3 MNIST AND DUSHION-MNIS DATASETS
The MNIST dataset (LeCun et al. 1998) consists of handwritten digits, and the FaSSIon-MNIST (FMNIST) dataset (Xiao al. 2017) is comprised of fashion products. Both the MNIST and Fashion- MNIST datasets have 60,000 training and 10,000 test 28 grayscale images. Each pixel was scaled range of 0-1. The datasets consist of 10 classes, and the number of data points per class is balanced. Table 12 shows the network architectures of SLOGAN used for the MNIST and FMNIST datasets. Conv k k. 8, n denotes convolutional network with n. feature maps, filter size k: x k and stride s. Deconv & x k, s, n denotes deconvolutional network with n feature maps, filter size k k., and stride s. For the MNIST dataset, we set 10. 7) = 0.0001, 7 0.002,
and m = 0.5. For MNIST-2, we set À = 4, = 0.0001, = 0.002, 4, and m = 0.5. For the For FMNIST-5, we set FMNIST dataset, we set À 10, 1) = 0.0001, 7 0.001, and A 1, 7) 0.0002, 7 0.004, = 4, and 0.5.
E.4 10K_73K DATASET
The 10x_73k dataset (Zeng et al. 2017) consists of 73,233 720-dimensional vectors, which are obtained from RNA transcript counts, and has eight cell types (classes). The number of data points per cell type is 10,085, 2.612, 9,232, 8,385, 10,224, 11,953, 10,479, and 10,263. We converted each element to logscale (i.e., log, (z + 1)) and scaled each element range between 0 and 1. Table shows the network architectures of SLOGAN used for the 10x 73k dataset. We set = 10, " 0.0001, 7 0.004, 4, and m 0.
31

---

Published as a conference paper at ICLR 2022
Table 12: SLOGAN architecture used for the MNIST and FMNIST datasets
D
E 28x
Z 1xx1x64
X 28x228x1 Conv 4x4, 64 + LReLU
Deconv 1, 1, 1024 BN + ReLU
Conv 4x4, 64 LReLU
Deconv 7x7, 1, 128 BN + ReLU
Conv 4x4, 64 LReLU
Conv 4x4, LReLU
Deconv 4x4, 2. 64 +
ReLU
Cony 7x7, 1, 1024 LReLU
Conv 7x7. 1024 LReLU
Deconv 4x4, 2, + Sigmoid
Conv 1x1, 1,
Conv 1x1,
Table 13: SLOGAN architecture used for the 10x_73k dataset
G
D
E
2 1244
X R
E R2
Linear 256 + LReLU
Linear 256 + LReLU
Linear 256 SN + LReLU
Linear 256 + LReLU
Linear 256 + LReLU
Linear 256 SN + LReLU
Linear 720
Linear 1
Linear 64 + SN
E.5 CIFAR-10 DATASET
The CIFAR-10 (rizhevsky et al. 2009) dataset comprises 50,000 training and 10,000 test 32x32 color images. Each pixel was scaled to range of to 1. The number of data points per class is balanced. Figure 23 and Table show the network architectures of residual blocks and SLOGAN used for the CIFAR datasets. AvgPool and GlobalAvgPool denote the average pooling and global average pooling layers, respectively. For the CIFAR-10, CIFAR-2, CIFAR-2 (7:3), and CIFAR-2 (9:1) datasets, we set À
0.0001, 7 0.002, 4, and 0.5.
Conv 3x3, 1, 128
AvgPool 2x2
Conv 3x3, 128
BN RoLU
Conv 3x3, 1, 128
Rel
Upsample 2xx
Upsample 2x2
RoLU
AvgPool 2x2
Conv 3x3, 128
Conv 3x3, 128
Conv 3x.3, 1, 128
Rel
BN ReL.U
ReLU
(a) Resblock
(b) Resblock Up
(c) Resblock Down
Figure 23: Resblock architectures used for colored image datasets.
Table 14: SLOGAN architecture used for the CIFAR datasets
D
E
8 E IR 128
X E x2xx3
X E 32x32x
Linear 2048 + Reshape 4, 4, 128
Resblock Down
Resblock Down
Resblock Up
Resblock Down
Resblock Down
Resblock Up
Resblock
Resbluck
Resblock Up
Resblock
Resblock
BN ReLU
ReLU GlobalAvgPool
ReLU GlobalAvgPool
Conv 3x3, 1, 3 Tanh
Linear
Linear 128
E.6 CELEBA DATASET
The CelebA dataset (Luu et al. 2015) consists of 202,599 face attributes. We cropped the face part of each image to x40x 140 pixels, resized it to 64 pixels, and sealed it to range between -1 and 1.
32

---

Published as a conference paper at ICLR 2022
The imbalanced ratio is different for each attribute, and the imbalanced ratios of male and eyeglasses used in the experiment are 1:1 and 14:1, respectively. Table [5 lists the network architectures of SLOGAN used for the CelebA dataset. We set À = 1, 17 0.0002, y 0.0006, and 0.5.
Table 15: SLOGAN architecture used for the CclebA dataset
E
E IR 128
X E 666666
X R61x6EX3
Linear 8192 + Reshape 8, 8, 128
Resblock Down
Resblock Down
Resblock Up
Resblock Down
Resblock Down
Resblock Up
Resblock Down
Resblock Down
Resblock Up
Resblock
Resblock
BN ReLU
ReLU GlobalAvgPuol
ReLU GlobalAvgPool
Conv 3x3, 1, 3 Tanh
Linear
Linear 128
E.7 CELEBA-HQ DATASET
The CelehA-HQ dataset KKrras et al. 2017) consists of 30,000 face attributes. We resized each image 128x 128 and 256x256 pixels, and scaled it range between -1 and 1. The imbalance ratio of male used in the experiment was 1.7:1. We used StyleGAN2 (Karras et al., (220)) architecture with DiffAugment and applied implicit reparameterization to the input space of the mapping network. We set À = 1, 1] = 0.002, 7 0.0006, 1, and m 0.
E.8 AFHQ DATASET
The AFHQ dataset (Choi et al. 2020) consists of 15,000 high-quality animal faces. We used cats and dogs, and there are about 5,000 images each in the dataset. We resized each image to 256x pixels, and scaled it to a range between and 1. We set the imbalance ratios of cats and dogs to 1:1, 1:2, and 1:5 by reducing the number of cats in the training dataset. We used the same model architecture and hyperparameters as for the CelebA-HQ dataset.
E.9 CODE AVAILABILITY
Code is available httppp///gg..uu...mm/etaa//r///a
Di wewpuuunnnntstyyegeeee htap:///e........ooaaaaaaaaeaeer
33

---

Published as a conference paper at ICLR 2022
REFERENCES IN APPENDIX
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large-scale machine learning. In /2th USENIX symposium on operating systems design and implementation (OSDI 16). pp. 263-283, 2016.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pp. 214 223. PMLR, 2017.
Mohammadreza Arnandpour, Ali Sadeghian, Chunyuan Li, and Mingyuan Zhou. Partition- guided gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5099-5109, 2021.
Mohamed Ishmael Belghazi, Sai Rajeswar, Olivier Mastropictro, Negar Rostamzadeh, Jovana Mitrovic, and Aaron Courville. Hierarchical adversarially Iearned inference. arXiv preprint arXiv:1802.01071, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence. 35(8):1798-1828. 2013.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In Neural Information Processing Systems (NIPS), 2016.
Jacob Cohen. Statistical power analysis for the behavioral sciences. Routledge, 2013.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4690-4699, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In 5th Interna- tional Conference on Learning Representations, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martín Arjovsky, Olivier Mastropictro, and Aaron C. Courville. Adversarially learned inference. In 5th International Conference Learning Representations, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Proceedings of the 3 International Conference Neural Information Processing Systems, pp. 6629-6640, 2017.
Ayush Jaiswal, Wael AbdAlmageed, Yue Wu, and Premkumar Natarajan. Bidirectional conditional generative adversarial networks. In Asian Conference on Computer Vision, pp. 216-232. Springer, 2018.
Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. In Advances in Neural Information Processing Systems, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X1v:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009,
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):227-2333 1998.
Wu Lin, Mohammad Emtiyaz Khan, and Mark Schmidt. Stein's lemma for the reparameterization trick with exponential family mixtures. arXiv preprint arXiv: 13398, 2019.
34

---

Published as a conference paper at ICLR 2022
Steven Liu, Tongzhou Wang. David Bau, Jun- Yan Zhu, and Antonio Torralba. Diverse image generation via self-conditioned gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14286-14295, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Deepak Mishra, Aravind Jayendran, and AP Prathosh. Effect of the latent structure on clustering with gans. IEEE Signal Processing Letters, 27:900-904, 2020.
Takeru Miyato and Masanori Koyama. egans with projection discriminator. In International Conference Learning Representations, 2018
Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan. Clustergan: Latent space clustering in generative adversarial networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4610-4617, 2019.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference machine learning, pp. 2642-2651. PMLR, 2017.
Lili Pan, Peijun Tang, Zhiyong Chen, and Zenglin Xu. Contrastive disentanglement in generative adversarial networks. arXiv preprint arXiv:2103.03638 2021.
Henning Petzka, Asja Fischer, and Denis Lukovnikov. On the regularization of wasserstein GANs. In International Conference on Learning Representations, 2018.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? Proceedings of the European Conference on Computer Vision (ECCV), pp. 213-229, 2018.
Dávid Terjék. Adversarial lipschitz regularization. In International Conference Learning Repre- sentations, 2020.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning classify images without labels. In European Conference Computer Vision, pp. 268-285. Springer, 2020.
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics, pp. 196-202. Springer, 1992.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1r08v:1708.07747 2017.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Shengyu Zhao, Zhijian Liu, Ji Lin. Jun-Yan Zhu, and Song Han. Differentiable augmentation for
data-efficient gan training. Advances in Neural Information Processing Systems, 33, 2020.
Grace XY Zheng, Jessica M Terry, Phillip Belgrader, Paul Ryvkin, Zachary W Bent, Ryan Wilson, Solongo B Ziraldo, Tobias D Wheeler, Geoff P McDermott, Junjie Zhu, et al. Massively parallel digital transcriptional profiling of single cells. Nature communications, 8(1):1-12, 2017.
35


Reviews:
[{"review_id": "J7MZYyv-mck", "reviewer": "Reviewer_5GSd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The idea of using Gaussian mixture as a prior for imbalanced attributes is intuitive and technically sound. \n2. In the theoretical aspect, this paper derives Stein latent optimization based on Stein’s lemma to estimate the gradient of parameters.\n3. The paper is well-written, and the result looks promising.\n\nWeaknesses:\n1. The imbalance datasets (MNIST and CIFAR) contain two classes. I would expect to evaluate over more imbalance settings of the datasets, such as one class is 0.1 fraction of the other nine classes. \n2. Two important baselines in GANs latent space clustering are missing. “Effect of The Latent Structure on Clustering with GANs” and “Mixture of GANs for Clustering”.\n3. Compared with ClusterGAN, this paper address the importance of generated image quality. The study may provide a comparison to Self-conditioned GAN or IC-GAN in terms of both generated image quality and clustering quality. \n4. In the current version, it is difficult to infer how U2C loss helps the performance of the proposed model. I would expect the study presents the quantitative results (e.g., Tables 1 and 2) of the proposed without U2C loss.\n\n*****************************\n\nPost Rebuttal Comments:\nThe authors have addressed my concerns in the revised version. I am satisfied with the response and change my rating from 5 to 6.\n", "summary_of_the_paper": "This paper uses Gaussian mixture as a prior to address the scenario of imbalanced attributes in GANs latent space clustering. The study derives Stein latent optimization that provides reparameterizable gradient estimations when assuming a Gaussian mixture prior for the latent space. The results show that the proposed model achieves superior performance compared to the baselines. ", "summary_of_the_review": "Although the derived Stein latent optimization for imbalanced attributes is interesting and promising, the current experiments are not enough to support the claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6lLDkY3kLtN", "reviewer": "Reviewer_bra9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. It is interesting to introduce Stein’s lemma to provide a first-order gradient identity for multivariate Gaussian distribution.\n2. The way of using a reparameterization trick to estimate gradients of the parameters of Gaussian mixtures is quite neat.\n3. The paper is well-written and clearly presented.\n\nWeakness:\n1. It is reasonable to conduct visualization comparisons with the most recent methods. \n2. The authors could present failure cases that may be associated with different attributes.\n3. How to obtain a suitable attribute ratio for the probe data? What will happen if we change the ratio from 14:1 to 1:14 for eyeglasses?", "summary_of_the_paper": "The paper proposed a Stein Latent Optimization for Generative adversarial networks, which can perform conditional generation in an unsupervised manner. The core innovation is the reparameterizable gradient estimations. The proposed unsupervised conditional contrastive loss further ensures the single attribute generation capacity. The idea is somewhat novel and motivated. The authors perform empirical studies on diverse datasets to conclude that the proposed algorithm is effective in learning balanced or imbalanced attributes.", "summary_of_the_review": "Overall, the paper is somewhat novel and easy to follow. The assumptions and decisions are well supported. The stepwise experiments are helpful and provide good insights to evaluate the proposed algorithm.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0JC6kQ-_uF-", "reviewer": "Reviewer_VThV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is fairly organized, overall supported by theoretical foundations and setups the right experiments for evaluating unsupervised conditional generation GANs.\n\nHowever, I find some of the paper's claims are neither trivial (or well-accepted IMO) nor are they supported/backed in the paper:\n\nFor example, the abstract says \"However, existing unsupervised conditional GANs cannot cluster attributes of these data in their latent spaces properly because they assume uniform distributions of the attributes.\" This claim needs to be supported by some reasoning/intuition and evidence/references as to why this is true. I can argue that this is not necessarily true, for example, [Self-Cond-Gan](https://openaccess.thecvf.com/content_CVPR_2020/papers/Liu_Diverse_Image_Generation_via_Self-Conditioned_GANs_CVPR_2020_paper.pdf), shows that clustering is possible using D's features in an unsupervised manner.\n\nSome of the novelties and tricks described in the paper (e.g. the reparametrization trick, or derivation and usage of the Theorem 1,2 are fairly trivial. However, the application of these ideas to Unsup GANs is a novel contribution of the paper.\n\nThe experiments are well designed, however, some of the right baselines have not been compared against. \nThe two models: \n\n1- [Self-Cond-Gan](https://openaccess.thecvf.com/content_CVPR_2020/papers/Liu_Diverse_Image_Generation_via_Self-Conditioned_GANs_CVPR_2020_paper.pdf) which is an unsupervised conditional GAN, based on the DCGAN architecture, and \n2- [PGM GAN](https://openaccess.thecvf.com/content/CVPR2021/papers/Armandpour_Partition-Guided_GANs_CVPR_2021_paper.pdf) which is also a GAN for unsupervised conditional image generation and also uses contrastive clustering \n\nAre both missing in the comparisons, both in the literature review and empirical comparisons. \n\nSince these two works are not discussed, based on my understanding and comparing the empirical results, It seems that SLOGAN does not achieve a comparable clustering performance to either (in terms of NMI over CIFAR-10) nor in terms of over all generation quality (FID). It would also be interesting to see how the models compare in terms of the newly introduced ICFID.\n\n", "summary_of_the_paper": "The paper is concerned with unsupervised image generation using GANs. They devise a solution (SLOGAN) for addressing the problems current models have, especially when dealing with datasets with unbalanced features/attributes. In addition, to learn attributes from data and improve the conditional generation, it suggests using a new contrastive loss (U2C). \n\n", "summary_of_the_review": "In summary, the paper needs some adjustments before it is ready for publication. The related works need to be updated with the relevant works (Self-Cond-GAN and PGMGAN mentioned above). The experiments and empirical results need to be compared with these relevant works or discussed why the comparison might not be fair. \n\nEDIT: Given the proposed method is moderately novel and the newly presented comparisons after the discussions with authors, I increased my rating from 5 to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-FFkJiIsfS2", "reviewer": "Reviewer_9K6Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this approach to be quite reasonable, and it consists of components that work well together. While GANs with mixture latent spaces have been proposed before, I am not aware of a method that efficiently updates the prior components. Previous work has used the reparameterization trick to derive component-specific updates, so the novelty of this specific component in using implicit reparameterization is a bit low, but the benefit is still there. I thought the introduction of the contrastive loss was interesting, but I had some concerns about the diversity of generated samples when the encodings are encouraged to match just the mean vector, ignoring the covariances. In the experiments, it looks to perform quite well, but I wonder how this would perform with less tightly-clustered data (even 8 Gaussians but with larger covariances).  \n\nFrom the experiments, I mostly wanted to see how this method handles a large imbalance on high-dimensional data. I am somewhat satisfied with the CelebA-HQ results with a ratio of 1.7:1, but this seems a bit low and does not stress the model at all. I wonder where does the model start to struggle. Is 2:1 possible?  Or even 5:1? Figure 5 examines this a little, but eyeglass is an attribute with global features almost identical to no-eyeglasses, so I don't think it is very informative. \n\n-----------------------------------------\n### Post Rebuttal Comments\n\nI thank the authors for their additional efforts in addressing my concerns. I am satisfied with this response and I will be retaining my score. \n", "summary_of_the_paper": "This work proposes a method for performing unsupervised conditional generation from imbalanced attributes. The key components of this work are the following:\n\n* A (architecture agnostic) GAN with a GMM latent space, where we have one component per attribute in the data. \n\n* The parameters of the components $(\\mu, \\Sigma)$ are learned via implicit reparameterization, and leveraging stein's lemma to derive gradients for $\\mu$ and $\\Sigma$. \n\n* The components are encouraged to be distinct by introducing a contrastive objective. An encoder is used to obtain a low dimensional vector representation of a generated sample, and this sample is encouraged to be close to the mean vector of the corresponding component in the latent space. ", "summary_of_the_review": "I think this is a reasonable combination of components that make progress on the task: facilitating learning of samplers from data with highly imbalanced class attributes. Given the performance increase compared to [1], it looks like using implicit reparameterization and a contrastive loss (the main novelty here) make a clear difference in performance. I think each of the components are justified and perform well. \n\n[1] Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 166–174, 2017.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an unsupervised learning method for GANs, called SLOGAN, which allows conditional generation of samples, by utilizing clustering structures of training data in a latent space. The main significance of the proposal over existing unconditional conditional GANs is that it is capable of dealing with training data with imbalance in the latent space. The proposal consists of the use of implicit reparameterization based on the generalized Stein lemma, which makes learning of the mixing coefficient parameters possible, as well as introduction of the U2C loss.

The initial review score distribution is such that two of them are just above the acceptance threshold, and two others are just below it. Upon reading the review comments and the author responses, as well as the paper itself, I think that the evaluations of the reviewers are more or less coherent with each other:

1. The proposed method is moderately, if not significantly, novel: The differences from DeLiGAN are the use of implicit reparameterization based on the generalized Stein lemma, learning of the mixing coefficient parameters, and introduction of the U2C loss.
2. The experimental results, while demonstrating effectiveness of the proposed method to some extent, were not convincing enough.

As for the item 2, the authors have provided results of additional experiments in their responses, as suggested by the reviewers, and two reviewers have revised their scores upward accordingly.

Yet another point I would like to mention is that in some numerical results summarized in Tables 1 and 2, as well as in several other places, one can notice somewhat large errors, so that one might be able to question the statistical significance of the claimed best-performing methods, shown in bold. (If my guess would be correct, the authors regarded the *best in the mean* as the best, ignoring the standard error, and did not perform any statistical testing to confirm the significance.) I would therefore appreciate additional assessment of significance of the numerical results via proper statistical testing.

Because of the above, I would like to recommend acceptance of this paper.
Decision: Accept (Poster)