STEIN LATENT OPTIMIZATION FOR GENERATIVE ADVERSARIAL NETWORKS
Year: 2022

Abstract:
Generative adversarial networks (GANs) with clustered latent spaces can perform conditional generation in a completely unsupervised manner. In the real world, the salient attributes of unlabeled data can be imbalanced. However, most of existing unsupervised conditional GANs cannot cluster attributes of these data in their latent spaces properly because they assume uniform distributions of the attributes. To address this problem, we theoretically derive Stein latent optimization that provides reparameterizable gradient estimations of the latent distribution parameters assuming a Gaussian mixture prior in a continuous latent space. Structurally, we introduce an encoder network and novel unsupervised conditional contrastive loss to ensure that data generated from a single mixture component represent a single attribute. We confirm that the proposed method, named Stein Latent Optimization for GANs (SLOGAN), successfully learns balanced or imbalanced attributes and achieves state-of-the-art unsupervised conditional generation performance even in the absence of attribute information (e.g., the imbalance ratio). Moreover, we demonstrate that the attributes to be learned can be manipulated using a small amount of probe data.

heading: 1 INTRODUCTION
text: GANs have shown remarkable results in the synthesis of realistic data conditioned on a specific class (Odena et al., 2017; Miyato & Koyama, 2018; Kang & Park, 2020). Training conditional GANs requires a massive amount of labeled data; however, data are often unlabeled or possess only a few labels. For unsupervised conditional generation, the salient attributes of the data are first identified by unsupervised learning and used for conditional generation of data. Recently, several unsupervised conditional GANs have been proposed (Chen et al., 2016; Mukherjee et al., 2019; Pan et al., 2021; Armandpour et al., 2021). By maximizing a lower bound of mutual information between latent codes and generated data, they cluster the attributes of the underlying data distribution in their latent spaces. These GANs achieve satisfactory performance when the salient attributes of data are balanced.
However, the attributes of real-world data can be imbalanced. For example, in the CelebA dataset (Liu et al., 2015), examples with one attribute (not wearing eyeglasses) outnumber the other attribute (wearing eyeglasses). Similarly, the number of examples with disease-related attributes in a biomedical dataset might be miniscule (Hwang et al., 2019). Thus, the imbalanced nature of real-world attributes must be considered for unsupervised conditional generation. Most of existing unsupervised conditional GANs are not suitable for real-world attributes, because they assume balanced attributes if the imbalance ratio is unknown (Chen et al., 2016; Mukherjee et al., 2019; Pan et al., 2021). Examples where existing methods fail to learn imbalanced attributes are shown in Figure 1 (a), (b) and (c).
In this paper, we propose unsupervised conditional GANs, referred to as Stein Latent Optimization for GANs (SLOGAN). We define the latent distribution of the GAN models as Gaussian mixtures to enable the imbalanced attributes to be naturally clustered in a continuous latent space. We derive reparameterizable gradient identities for the mean vectors, full covariance matrices, and mixing coefficients of the latent distribution using Stein’s lemma. This enables stable learning and makes
∗Corresponding author
9/21
latent distribution parameters, including the mixing coefficient, learnable. We then devise a GAN framework with an encoder network and an unsupervised conditional contrastive loss (U2C loss), which can interact well with the learnable Gaussian mixture prior (Figure 2). This framework facilitates the association of data generated from a Gaussian component with a single attribute.
For the synthetic dataset, our method (Figure 1 (d)) shows superior performance on unsupervised conditional generation, with the accurately learned mixing coefficients. We performed experiments on various real-world datasets including MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), CIFAR-10 (Krizhevsky et al., 2009), CelebA (Liu et al., 2015), CelebA-HQ (Karras et al., 2017), and AFHQ (Choi et al., 2020) using architectures such as DCGAN (Radford et al., 2016), ResGAN (Gulrajani et al., 2017), and StyleGAN2 (Karras et al., 2020). Through experiments, we verified that the proposed method outperforms existing unsupervised conditional GANs in unsupervised conditional generation on datasets with balanced or imbalanced attributes. Furthermore, we confirmed that we could control the attributes to be learned when a small set of probe data is provided.
The contributions of this work are summarized as follows:
• We propose novel Stein Latent Optimization for GANs (SLOGAN). To the best of our knowledge, this is one of the first methods that can perform unsupervised conditional generation by considering the imbalanced attributes of real-world data.
• To enable this, we derive the implicit reparameterization for Gaussian mixture prior using Stein’s lemma. Then, we devise a GAN framework with an encoder and an unsupervised conditional contrastive loss (U2C loss) suitable for implicit reparameterization.
• SLOGAN significantly outperforms the existing methods on unsupervised learning tasks, such as cluster assignment, unconditional data generation, and unsupervised conditional generation, on datasets that include balanced or imbalanced attributes.

heading: 2 BACKGROUND

heading: 2.1 GENERATIVE ADVERSARIAL NETWORKS
text: Unsupervised conditional generation Several models including InfoGAN (Chen et al., 2016), ClusterGAN (Mukherjee et al., 2019), Self-conditioned GAN (Liu et al., 2020), CD-GAN (Pan et al., 2021), and PGMGAN (Armandpour et al., 2021) have been proposed to perform conditional generation in a completely unsupervised manner. However, these models primarily have two drawbacks: (1) Most of these methods embed the attributes in discrete variables, which induces discontinuity among the embedded attributes. (2) Most of them assume uniform distributions of the attributes, and thus fail to learn the imbalance in attributes when the imbalance ratio is not provided. In Appendix B.2, we discuss the above models in detail. By contrast, our work addresses the aforementioned limitations by combining GANs with the gradient estimation of the Gaussian mixture prior via Stein’s lemma and representation learning on the latent space.
5
Method
GANs with Gaussian mixture prior DeLiGAN (Gurumurthy et al., 2017) is analogous to the proposed method, as it assumes a Gaussian mixture prior and learns the mean vectors and covariance matrices via the reparameterization trick. However, DeLiGAN assumes uniform mixing coefficients without updating them. As a result, it fails to perform unsupervised conditional generation on datasets with imbalanced attributes. In addition, it uses the explicit reparameterization trick, which inevitably suffers from high variance in the estimated gradients. This will be discussed further in Section 2.3.

heading: 2.2 CONTRASTIVE LEARNING
text: Contrastive learning aims to learn representations by contrasting neighboring with non-neighboring instances (Hadsell et al., 2006). In general, contrastive loss is defined as a critic function that approximates the log density ratio log p(y|x)/p(y) of two random variables X and Y . By minimizing the loss, the lower bound of the mutual information I(X;Y ) is approximately maximized (Poole et al., 2019). Several studies have shown that contrastive losses are advantageous for the representation learning of imbalanced data (Kang et al., 2021; 2020; Wanyan et al., 2021). Motivated by these observations, we propose a contrastive loss that cooperates with a learnable latent distribution.

heading: 2.3 GRADIENT ESTIMATION FOR GAUSSIAN MIXTURE
text: Stein’s lemma Stein’s lemma provides a first-order gradient identity for a multivariate Gaussian distribution. The univariate case of Stein’s lemma can be described as follows: Lemma 1. Let function h(·) : R 7→ R be continuously differentiable. q(z) is a univariate Gaussian distribution parameterized by the mean µ and variance σ. Then, the following identity holds:
Eq(z) [ σ−1(z − µ)h(z) ] = Eq(z) [∇zh(z)] (1)
Lin et al. (2019b) generalized Stein’s lemma to exponential family mixtures and linked it to the implicit reparameterization trick. Stein’s lemma has been applied to various fields of deep learning, including Bayesian deep learning (Lin et al., 2019a) and adversarial robustness (Wang et al., 2020). To the best of our knowledge, our work is the first to apply Stein’s lemma to GANs.
Reparameterization trick A simple method to estimate gradients of the parameters of Gaussian mixtures is explicit reparameterization, used in DeLiGAN. When the c-th component is selected according to the mixing coefficient p(c), the latent variable is calculated as follows: z = µc+ϵ ·Σ1/2c , where ϵ ∼ N (0, I). Derivatives of a loss function ∂L(z)∂µ and ∂L(z) ∂Σ only update the mean and covariance matrices of the selected (c-th) component µc and Σc, respectively. Gradient estimation using explicit reparameterization is unbiased; however, it has a distinctly high variance. For a single latent vector z, the implicit reparameterization trick (Figurnov et al., 2018) updates the parameters of all the latent components. Gradient estimation using implicit reparameterization is unbiased and has a lower variance, which enables a more stable and faster convergence of the model. The gradients for the parameters of the Gaussian mixture prior in our method are implicitly reparameterizable.

heading: 3 PROPOSED METHOD
text: In the following sections, we propose Stein Latent Optimization for GANs (SLOGAN). We assume a Gaussian mixture prior (Section 3.1), derive implicit reparameterization of the parameters of the mixture prior (Section 3.2), and construct a GAN framework with U2C loss (Section 3.3). Additionally, we devise a method to manipulate attributes to be learned if necessary (Section 3.4). An overview of SLOGAN is shown in Figure 2.

heading: 3.1 GAUSSIAN MIXTURE PRIOR
text: We consider a GAN with a generator G : Rdz 7→ Rdx and a discriminator D : Rdx 7→ R, where dz and dx are the dimensions of latent and data spaces, respectively. In the latent space Z ∈ Rdz , we consider a conditional latent distribution q(z|c) = N (z;µc,Σc), c = 1, ...,K, where K is the number of components we initially set and µc, Σc are the mean vector and covariance matrix of the cth component, respectively. Subsequently, we consider a Gaussian mixture q(z) = ∑K c=1 p(c)q(z|c) parameterized by µ = {µc}Kc=1, Σ = {Σc}Kc=1 and π = {πc}Kc=1 = {p(c)}Kc=1 as the prior. We hypothesize that a mixture prior in a continuous space could model some continuous attributes of real-world data (e.g., hair color) more naturally than categorical priors which could introduce discontinuity (Mukherjee et al., 2019). Because we use implicit reparameterization of a mixture of Gaussian priors (derived in Section 3.2), SLOGAN can fully benefit from implicit reparameterization and U2C loss. By contrast, the implicit reparameterization of prior distributions that do not belong to the exponential family (e.g., categorical priors) remains an open question.
In the experiments, the elements of µc were sampled from N (0, 0.1), and we selected Σc = I and πc = 1/K as the initial values. For the convenience of notation, we define the latent distribution q = q(z), the mixing coefficient πc = p(c), and δ(z) = {δ(z)c}Kc=1, where δ(z)c = q(z|c)/q(z). q(c|z), the responsibility of component c for a latent vector z, can be expressed as follows:
q(c|z) = q(c, z) q(z) = q(z|c)p(c) q(z) = δ(z)cπc (2)

heading: 3.2 GRADIENT IDENTITIES
text: We present gradient identities for the latent distribution parameters. To derive the identities, we use the generalized Stein’s lemma for Gaussian mixtures with full covariance matrices (Lin et al., 2019b). First, we derive a gradient identity for the mean vector using Bonnet’s theorem (Bonnet, 1964). Theorem 1. Given an expected loss of the generator L and a loss function for a sample ℓ(·) : Rdz 7→ R, we assume ℓ to be continuously differentiable. Then, the following identity holds:
∇µcL = Eq [δ(z)cπc∇zℓ(z)] (3)
Proof of Theorem 1 is given in Appendix C.1.
We derive a gradient identity for the covariance matrix via Price’s theorem (Price, 1958). Among the two versions of the Price’s theorem, we use the first-order identity to minimize computational cost. Theorem 2. With the same assumptions as in Theorem 1, the following gradient identity holds:
∇ΣcL = 1 2 Eq [ δ(z)cπcΣ −1 c (z− µc)∇Tz ℓ(z) ] (4)
Proof of Theorem 2 is given in Appendix C.2. In the implementation, we replaced the expectation of the right-hand side of Equation 4 with the average for a batch of latent vectors; hence, the updated Σc may not be symmetric or positive-definite. To force a valid covariance matrix, we modify the updates of the covariance matrix as follows:
∆Σc = −∇ΣcL = − 1 2 Eq [ 1 2 ( Sz + S T z )] (5)
∆Σ′c = ∆Σc + γ
2 ∆ΣcΣ
−1 c ∆Σc (6)
Algorithm 1 Training procedure of SLOGAN Initialize µ, Σ, ρ, parameters of D, G, and E while training loss is not converged do
Sample a batch of data {xi}Bi=1 ∼ p(x) Sample a batch of latent vectors {zi}Bi=1 ∼ q(z) for i = 1, ..., B do
Calculate ℓadv(zi) and ℓU2C(zi) for a latent vector zi
Szi ← δ(zi)cπcΣ−1c (zi−µc)∇Tzi(ℓadv(zi)+λℓU2C(zi)) end for for c = 1, ...,K do
Update µc, Σc and ρc via stochastic gradient estimation µc ← µc−γ 1B ∑B i=1 δ(z i)cπc∇zi(ℓadv(z i)+λℓU2C(z i))
∆Σc ← − 14B ∑B i=1 ( Szi + S T zi ) Σc ← Σc + γ ( ∆Σc +
γ 2 ∆ΣcΣ −1 c ∆Σc ) ρc ← ρc − γ 1B ∑B i=1 πc ( δ(zi)c − 1 ) ℓadv(z
i) end for Update G, E and D using SGD ∇G,E 1B ∑B i=1 ( ℓadv(z i) + λℓU2C(z i) )
∇D ( − 1
B ∑B i=1 ℓadv(z i)− 1 B ∑B i=1 D(x i) )
end while
Algorithm 2 Intra-cluster FID input :{{xiy}Ni=1}Ky=1 - Data sampled from
p(x|y) for y = 1, ...,K; {{zic}Ni=1}Kc=1 - Latent vectors sampled from q(z|c) for c = 1, ...,K
output :ICFID - Intra-cluster FID; Yc - Class-cluster assignments
Y ← {1, ...,K} C ← {1, ...,K} for each class y in Y do
Xr ← {xiy}Ni=1 for each cluster c in C do
Xg ← {xic}Ni=1 d(y, c)← FID(Xr,Xg)
end for c∗ ← argminc∈C d(y, c) ICFID(y)← d(y, c∗) Yc(y)← c∗ Remove c∗ from C
end for ICFID← 1
K ∑K y=1 ICFID(y)
where Sz = δ(z)cπcΣ−1c (z− µc)∇Tz ℓ(z), and γ denotes the learning rate for Σc. Equation 5 holds as ∆Σc = 12Eq [Sz] = 1 2Eq [ STz ] . Motivated by Lin et al. (2020), Equation 6 ensures the positive-definiteness of the covariance matrix, which is proved by Theorem 3. Theorem 3. The updated covariance matrix Σ′c = Σc+γ∆Σ′c with the modified update rule specified in Equation 6 is positive-definite if Σc is positive-definite.
Proof of Theorem 3 is provided in Appendix C.3.
We introduce a mixing coefficient parameter ρc, which is updated instead of the mixing coefficient πc, to guarantee that the updated mixing coefficients are non-negative and summed to one. πc can be calculated using the softmax function (i.e., πc = exp(ρc)/ ∑K i=1 exp(ρi)). We can then derive the gradient identity for the mixing coefficient parameter as follows: Theorem 4. Let ρc be a mixing coefficient parameter. Then, the following gradient identity holds:
∇ρcL = Eq [πc (δ(z)c − 1) ℓ(z)] (7)
Proof of Theorem 4 is given in Appendix C.4. Because the gradients of the latent vector with respect to the latent parameters are computed by implicit differentiation via Stein’s lemma, we obtain the implicit reparameterization gradients introduced by Figurnov et al. (2018).

heading: 3.3 CONTRASTIVE LEARNING
text: We introduce new unsupervised conditional contrastive loss (U2C loss) to learn salient attributes from data and to facilitate unsupervised conditional generation. We consider a batch of latent vectors {zi}Bi=1 ∼ q(z), where B is the batch size. Generator G receives the i-th latent vector zi and generates data xig = G(z i). The adversarial loss for G with respect to the sample zi is as follows:
ℓadv(z i) = −D(G(zi)) (8)
We also introduce an encoder network E to implement U2C loss. The synthesized data xig enters E, and E generates an encoded vector eix = E(x i g). Then, we find the mean vector µ i C , where C is the component ID with the highest responsibility q(c|zi). We calculate C first because a generated sample should have the attribute of the most responsible component among multiple components in the continuous space. Second, to update the parameters of the prior using implicit reparameterization, the loss should be a function of a latent vector zi, as proved in Theorems 1, 2, and 4. The component ID for each sample is calculated as follows:
Ci = argmax c q(c|zi) = argmax c δ(zi)cπc (9)
where q(c|zi) = δ(zi)cπc is derived from Equation 2. To satisfy the assumption of the continuously differentiable loss function in Theorems 1 and 2, we adopt the Gumbel-Softmax relaxation (Jang et al., 2017), instead of the argmax function. We use µiC = ∑K c=1 C i cµc to calculate U2C loss to ensure that the loss function is continuously differentiable with respect to zi, where Ci = Gumbel-Softmaxτ (δ(z i)π) and τ = 0.01. We derive U2C loss as follows:
ℓU2C(z i) = − log exp(cos θii)
1 B ∑B j=1 exp(cos θij)
(10)
where we select the cosine similarity between eix and µ j C, cos θij = e i x ·µ j C/∥eix∥∥µ j C∥ as the critic function that approximates the log density ratio log p(Cj |xig)/p(Cj) for contrastive learning. Given a test data, the probability for each cluster can be calculated using the assumption of the critic function, which enables us to assign a cluster for the data. Cluster assignment is described in Appendix D.2.
Intuitively, a mean vector µiC of a latent mixture component is regarded as a prototype of each attribute. U2C loss encourages the encoded vector eix of the generated sample to be similar to their assigned low-dimensional prototypes µiC in the latent space. This allows each salient attribute clusters in the latent space, and each component of the learned latent distribution is responsible for a certain attribute of the data. If cos θii is proportional to the log density ratio log p(Ci|xig)/p(Ci), minimizing U2C loss in Equation 10 is equivalent to maximizing the lower bound of the mutual information I(Ci;xig), as discussed by Poole et al. (2019) and Zhong et al. (2020).
G and E are trained to minimize 1B ∑B i=1 ( ℓadv(z i) + λℓU2C(z i) ) , where λ denotes the coefficient of U2C loss. Both µ and Σ are learned by substituting ℓadv(zi)+λℓU2C(zi) into ℓ of Equations 3 and 6, respectively. When U2C loss is used to update π, U2C loss hinders π from estimating the imbalance ratio of attributes in the data well, which is discussed in Appendix A.3 with a detailed explanation and an empirical result. Therefore, ρ, from which π is calculated, uses only the adversarial loss, and ℓ of Equation 7 is substituted by ℓadv(zi). µ, Σ and ρ are learned using a batch average of estimated gradients, which is referred to as stochastic gradient estimation, instead of expectation over the latent distribution q. The entire training procedure of SLOGAN is presented in Algorithm 1.
To help that the latent space does not learn low-level attributes, such as background color, we additionally used the SimCLR (Chen et al., 2020) loss on the generated data with DiffAugment (Zhao et al., 2020) to train the encoder on colored image datasets. Methodological details and discussion on SimCLR are presented in Appendix D.4 and A.3, respectively.

heading: 3.4 ATTRIBUTE MANIPULATION
text: For datasets such as face attributes, a data point can have multiple attributes simultaneously. To learn a desired attribute from such data, a probe dataset {xic}Mi=1 for the c-th latent component, which consists of M data points with the desired attribute, can be utilized. We propose the following loss:
Lm = 1
M M∑ i=1 − log exp(cos θ i c)∑K k=1 exp(cos θ i k)
(11)
where cos θik = E(x i c) · µk/∥E(xic)∥∥µk∥ is the cosine similarity between E(xic) and µk. Our model manipulates attributes by minimizing Lm for µ,Σ, G, and E. In addition, mixup (Zhang et al., 2018) can be used to better learn attributes from a small probe dataset. The advantage of SLOGAN in attribute manipulation is that it can learn imbalanced attributes even if the attributes in the probe dataset are balanced, and perform better conditional generation. The detailed procedure of attribute manipulation is described in Appendix D.3.

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS
text: We used the MNIST (LeCun et al., 1998), Fashion-MNIST (FMNIST) (Xiao et al., 2017), CIFAR-10 (Krizhevsky et al., 2009), CelebA (Liu et al., 2015), CelebA-HQ (Karras et al., 2017), and AFHQ (Choi et al., 2020) datasets to evaluate the proposed method. We also constructed some datasets with
imbalanced attributes. For example, we used two classes of the MNIST dataset (0 vs. 4, referred to as MNIST-2), two classes of the CIFAR-10 dataset (frogs vs. planes, referred to as CIFAR-2), and five clusters of the FMNIST dataset ({Trouser}, {Bag}, {T-shirt/top, Dress}, {Pullover, Coat, Shirt}, {Sneaker, Sandal, Ankle Boot}, referred to as FMNIST-5 with an imbalance ratio of 1:1:2:3:3). Details of the datasets are provided in Appendix E.
Although SLOGAN and other methods do not utilize labels for training, the data in experimental settings have labels predefined by humans. We consider that each class of dataset contains a distinct attribute. Thus, the model performance was measured using classes of datasets. The number of latent components or the dimension of the discrete latent code (K) was set as the number of classes of data.

heading: 4.2 EVALUATION METRICS
text: The performance of our method was evaluated quantitatively in three aspects: (1) whether the model could learn distinct attributes and cluster real data (i.e., cluster assignment), which is evaluated using normalized mutual information (NMI) (Mukherjee et al., 2019), (2) whether the overall data distribution p(xr) could be estimated (i.e., unconditional data generation), which is measured using the Fréchet inception distance (FID) (Heusel et al., 2017), and, most importantly, (3) whether the data distribution for each attribute p(xr|c) could be estimated (i.e., unsupervised conditional generation). For unsupervised conditional generation, it is important to account for intra-cluster diversity as well as the quality of the generated samples. We introduce a modified version of FID named intra-cluster Fréchet inception distance (ICFID) described in Algorithm 2. We calculate FIDs between the real data of each class and generated data from each latent code (a mixture component for DeLiGAN and SLOGAN, and a category for other methods). We then greedily match a latent code with a class of real data with the smallest FID. We define ICFID as the average FID between the matched pairs and use it as an evaluation metric for unsupervised conditional generation. ICFID additionally provides class-cluster assignment (i.e., which cluster is the closest to the class).
Ratio of planes (10-𝑥 : 𝑥)
N M
I ↑
Ratio of planes (10-𝑥 : 𝑥)
IC F
ID ↓
SLOGAN ClusterGAN CD-GAN (a) FMNIST-5 (1:1:2:3:3) (b) CIFAR-2 (7:3)
(a) (b)
Figure 4: Performance comparison with respect to the imbalance ratio on (a) cluster assignment and (b) unsupervised conditional generation.
Table 3: Effectiveness of U2C loss
Dataset Ablation ICFID ↓
CIFAR-10 SLOGAN w/o ℓU2C 78.26 SLOGAN 71.23
MNIST-2 SLOGAN w/o ℓU2C 9.43 (7:3) SLOGAN 5.91
Synthetic SLOGAN w/o ℓU2C ✗ SLOGAN ✓

heading: 4.3 EVALUATION RESULTS
text: We compared SLOGAN with InfoGAN (Chen et al., 2016), DeLiGAN (Gurumurthy et al., 2017), ClusterGAN (Mukherjee et al., 2019), Self-conditioned GAN (SCGAN) (Liu et al., 2020), CD-GAN (Pan et al., 2021), and PGMGAN (Armandpour et al., 2021). Following Mukherjee et al. (2019), we used k-means clustering on the encoder outputs of the test data to calculate NMI. DeLiGAN has no encoder network; hence the pre-activation of the penultimate layer of D was used for the clustering metrics. For a fair comparison, we also compared DeLiGAN with an encoder network (referred to as DeLiGAN+). The same network architecture and hyperparameters (e.g., learning rate) were used across all methods for comparison. Details of the experiments and DeLiGAN+ are presented in Appendices E and D.5, respectively.
Balanced attributes We compare SLOGAN with existing unsupervised conditional GANs on datasets with balanced attributes. As shown in Table 1 (The complete version is given in Appendix A.1.), SLOGAN outperformed other GANs, and comparisons with methods with categorical priors (ClusterGAN and CD-GAN) verified the advantages of the mixture priors.
Imbalanced attributes In Table 2 (The complete version is presented in Appendix A.2), we compare SLOGAN with existing methods on datasets with imbalanced attributes. ICFIDs of our method are much better than those of other methods, which indicates that SLOGAN was able to robustly capture the minority attributes in datasets and can generate data conditioned on the learned attributes. In CIFAR-2 (7:3), the ratio of frog and plane is 7 to 3 and the estimated π is (0.69±0.02, 0.31±0.02), which are very close to the ground-truth (0.7, 0.3). Figure 3 (a) shows the images generated from each latent component of SLOGAN on AFHQ (Cat:Dog=1:2). More qualitative results are presented in Appendix A.7.
Performance with respect to imbalance ratio We compared the performance of SLOGAN with competitive benchmarks (ClusterGAN and CD-GAN) by changing the imbalance ratios of CIFAR-2 from 9:1 to 1:9. SLOGAN showed higher performance than the benchmarks on cluster assignment (Figure 4 (a)) and unsupervised conditional generation (Figure 4 (b)) for all imbalance ratios. Furthermore, our method shows a larger gap in ICFID with the benchmarks when the ratio of planes is low. This implies that SLOGAN works robustly in situations in which the attributes of data are highly imbalanced. We conducted additional experiments including interpolation in the latent space, benefits of ICFID. The results of the additional experiments are shown in Appendix A.
(a) Male (1:1)
(b) Eyeglasses (14:1)

heading: 4.4 ABLATION STUDY
text: U2C loss Table 3 (The complete version is given in Appendix A.3) shows the benefit of U2C loss on several datasets. Low-level features (e.g., color) of the CIFAR dataset differ depending on the class, which enables SLOGAN to function to some extent without U2C loss on CIFAR-10. In the MNIST dataset, the colors of the background (black) and object (white) are the same, and only the shape of objects differs depending on the class. U2C loss played an essential role on MNIST (7:3). The modes of the Synthetic dataset (Figure 1) are placed adjacent to each other, and SLOGAN cannot function on this dataset without U2C loss. From the results, we observed that the effectiveness of U2C loss depends on the properties of the datasets.
Implicit reparameterization To show the advantage of implicit over explicit reparameterization, we implemented DeLiGAN with U2C loss by applying explicit reparameterization on µ and Σ. Because the mixing coefficient cannot be updated with explicit reparameterization to the best of our knowledge, we also implemented DeLiGAN with U2C loss and implicit reparameterization on ρ using Equation 7. In Table 4, SLOGAN using implicit reparameterization outperformed explicit reparameterization. When implicit ρ update was added, the prior collapsed into a single component (πy=0 = 1) and ICFID increased. The lower variance of implicit reparameterized gradients prevents the prior from collapsing into a single component and improves the performance. Additional ablation studies and discussions are presented in Appendix A.3.

heading: 4.5 EFFECTS OF PROBE DATA
text: CelebA + ResGAN We demonstrate that SLOGAN can learn the desired attributes using a small amount of probe data. Among multiple attributes which co-exist in the CelebA dataset, we chose Male (1:1) and Eyeglasses (14:1). We randomly selected 30 probe images for each latent component. πy=0 represents the learned mixing coefficient that correspond to latent components associated with faces without the attribute. As shown in Figure 5 and Table 5, we observed that SLOGAN learned the desired attributes. Additional experiments on attribute manipulation are presented in Appendix A.3.
CelebA-HQ + StyleGAN2 StyleGAN2 (Karras et al., 2020) differs from other GANs in that the latent vectors are used for style. Despite this difference, the implicit reparameterization and U2C loss can be applied to the input space of the mapping network. On the CelebA-HQ dataset, we used 30 male and 30 female faces as probe data. As shown in Figure 3 (b), SLOGAN successfully performed on high-resolution images and a recent architecture, even simultaneously with imbalanced attributes.

heading: 5 CONCLUSION
text: We have proposed a method called SLOGAN to generate data conditioned on learned attributes on real-world datasets with balanced or imbalanced attributes. We derive implicit reparameterization for the parameters of the latent distribution. We then proposed a GAN framework and unsupervised conditional contrastive loss (U2C loss). We verified that SLOGAN achieved state-of-the-art unsupervised conditional generation performance. In addition, a small amount of probe data helps SLOGAN control attributes. In future work, we will consider a principled method to learn the number and hierarchy of attributes in real-world data. In addition, improving the quality of samples with minority attributes is an important avenue for future research on unsupervised conditional GANs.

heading: ACKNOWLEDGEMENT
text: This work was supported by the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2021, Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.20210-01343, Artificial Intelligence Graduate School Program (Seoul National University)], and AIR Lab (AI Research Lab) in Hyundai Motor Company through HMC-SNU AI Consortium Fund.

heading: A ADDITIONAL RESULTS AND DISCUSSION
text: We also compared the proposed method with WGAN (Arjovsky et al., 2017) with the proposed method. We used the pre-activation of the penultimate layer of D for the clustering metrics because WGAN has no encoder network. We could not measure ICFID of WGAN because it cannot perform unsupervised conditional generation. In addition to various datasets, we also used the 10x 73k dataset (Zheng et al., 2017), which consists of RNA transcript counts. From the results of the clustering performances on the 10x 73k dataset, we show that SLOGAN learns useful imbalanced attributes and can be helpful in the use of unlabeled biomedical data.

heading: A.1 PERFORMANCE COMPARISON ON BALANCED ATTRIBUTES

heading: A.2 PERFORMANCE COMPARISON ON IMALANCED ATTRIBUTES

heading: A.3 ABLATION STUDY
text: Table 8 shows the ablation study on SLOGAN trained with CIFAR-2 (7:3). πy=0 and πy=1 represent the mixing coefficients of the latent components that correspond to the frogs and planes, respectively, and the ground-truth of πy=0 is 0.7.
Factor analysis Rows 1-6 of Table 8 compare the performance depending on the factors affecting the performance of SLOGAN (µ, Σ, ρ updates, and ℓU2C). We confirmed that SLOGAN with all the factors demonstrated the highest performance. Among the parameters of the latent distribution, the µ update leads to the highest performance improvement. The intra-cluster Fréchet inception distance (ICFID) of SLOGAN without ρ update (the 5th row of Table 8) indicates that SLOGAN outperformed existing unsupervised conditional GANs even when assuming a uniform distribution of the attributes.
Loss for ρ update We do not use U2C loss ℓU2C to learn the mixing coefficient parameters ρ. We construct U2C loss to approximate the negative mutual information −I(C;xg) that can be
decomposed into entropy and conditional entropy as follows:
ℓU2C(z) ≈ −I(C;xg) = H(C|xg)−H(C) (12)
The conditional entropy term reduces the uncertainty of the component from which the generated data are obtained. The entropy term promotes that component IDs are uniformly distributed. In terms of ρ update, the entropy term H(C) drives p(C) toward a discrete uniform distribution. Therefore, using ℓU2C for learning ρ pulls π to a discrete uniform distribution and can hinder the learned π from accurately estimating the imbalance ratio inherent in the data. In the 9th row of Table 8, we observed that the unsupervised conditional generation performance was undermined and the estimated imbalance ratio (πy=0) was learned closer to a discrete uniform distribution (0.5) when ℓU2C was used for ρ update.
SimCLR analysis For the colored image datasets, SLOGAN uses the SimCLR loss for the encoder with only fake (generated) data to further enhance the unsupervised conditional generation performance. The 10th to 12th rows of Table 8 show several ablation studies that analyzed the effect of SimCLR loss on SLOGAN. SLOGAN without SimCLR still showed at least approximately 35% performance improvement compared to the existing unsupervised conditional GANs (ICFID of
ClusterGAN: 75.52, CD-GAN: 76.91 in Table 2), even considering the fair computational cost and memory consumption. The SimCLR loss shows the highest performance improvement especially when applied only to fake data. SimCLR improved the performance by 7%.
Attribute manipulation As shown in the 13th and 14th rows of Table 8, probe data significantly improved the performance of SLOGAN on CIFAR-2 (7:3) with 10 probe data for each latent component. We also confirmed that the mixup applied to the probe data further enhanced the overall performance of our model. Figure 6 (a) shows the data generated from SLOGAN trained on CIFAR-2 (9:1) without the probe data. With extremely imbalanced attributes, SLOGAN mapped frog images with a white background onto the same component as airplanes in its latent space. When we use 10 probe data for each latent component, as shown in Figure 6 (b), frogs with a white background were generated from the same latent component as the other frog images.
Feature scale We introduced the feature scale s described in Appendix D.1 to reinforce the discriminative power of U2C loss. For the MNIST-2 (7:3) dataset, s was set to 4. Such a parameter configuration is justified by a greedy search in [0.5, 1, 2, 4, 8]. The performances of SLOGAN on MNIST-2 (7:3) with different feature scales are shown in Table 10.
Intuitively, increasing the feature scale s makes the samples generated from the same component closer to each other in the embedding space. From these results, we observed that the optimal choice of the temperature factor enhances the discriminative power of U2C loss.

heading: A.4 STATISTICAL SIGNIFICANCE
text: We completed the statistical tests between SLOGAN and other methods in Tables 1 and 2. Since the results of the experiment could not satisfy normality and homogeneity of variance, we used the Wilcoxon rank sum test (Wilcoxon, 1992). When p-value > 0.05, we measured the effect size using Cohen’s d (Cohen, 2013). We validated that all the experiments are statistically significant or showed large or medium effect sizes, with the exception of FID of InfoGAN vs. SLOGAN for FMNIST in Table 1, NMI of DeLiGAN+ vs. SLOGAN for FMNIST-5 in Table 2.
A.5 INTERPOLATION IN LATENT SPACE
We also qualitatively show that the continuous nature of the prior distribution of SLOGAN makes superbly smooth interpolation possible in the latent space. In Figure 7, we visualize images generated from latent vectors obtained via interpolation among the mean vectors of the trained latent components. The generated images gradually changed to 3, 5, and 8 for the MNIST dataset, and t-shirt/top, pullover, and dress for the Fashion-MNIST dataset. In particular, we confirmed that the face images generated from the model trained with the CelebA data changed smoothly. The continuous attributes of realworld data are well mapped to the continuous latent space assumed by us, unlike most other methods using separated latent spaces induced via discrete latent codes.

heading: A.6 BENEFITS OF ICFID
text: DeLiGAN and ClusterGAN trained on the MNIST-2 (7:3) exhibited comparable FIDs to SLOGAN (DeLiGAN: 4.21, ClusterGAN: 4.25, and SLOGAN: 4.02); however they showed ICFIDs approximately four times higher (DeLiGAN: 25.34, ClusterGAN: 25.61, and SLOGAN: 5.91). From the data generated from each latent component of DeLiGAN and ClusterGAN in Figure 8 (b) and (c), we confirm that the attributes were not learned well in the latent space of DeLiGAN. By contrast, from the data generated from each latent component of SLOGAN presented in Figure 8 (a), SLOGAN successfully learned the attributes in its latent space. This shows that ICFID is useful for evaluating the performance of unsupervised conditional generation. In addition, ICFID can evaluate the diversity of images generated from a discrete latent code or mode because ICFID is based on FID. As shown in Figure 9, when a mode collapse occurs, the diversity of samples decreases drastically, and DeLiGAN trained on the CIFAR-2 (7:3) shows approximately twice the ICFID than those of InfoGAN and ClusterGAN (DeLiGAN: 186.31, InfoGAN: 88.49, and ClusterGAN: 75:52).

heading: DeLiGAN

heading: A.7 QUALITATIVE RESULTS
text: Synthetic data Figures 10 and 11 shows the synthetic data and data generated by SLOGAN. Each color denotes generated data from each mixture component of the latent distribution. As the training progresses, each mixture component in the latent space is more strongly associated with each Gaussian distribution in the data space.
Generated images and latent spaces Figures 12, 13, 17, 18, 19, the left plot of Figure 15, and the upper plot of Figure 16 show the images generated from each latent component of SLOGAN trained on various datasets. Figure 14, the right plot of Figure 15, and the lower plot of Figure 16 visualize 1,000 latent vectors of SLOGAN trained on the MNIST, Fashion-MNIST, MNIST-2 (7:3), and FMNIST-5 datasets using 3D principal component analysis (PCA). Each color represents the component with the highest responsibility, and each image shows the generated image from the latent vector. As shown in Figure 14, similar attributes (e.g., 4, 7, and 9) are mapped to nearby components in the latent space.
Comparisons with the most recent methods We compare our method with the most recent methods such as CD-GAN (Pan et al., 2021) and PGMGAN (Armandpour et al., 2021) on the CIFAR-2 (7:3) dataset. From the results shown in Figure 20, we qualitatively confirm that SLOGAN learns imbalanced attributes of the dataset most robustly.
Highly imbalanced multi-class data We trained our method on highly imbalanced multi-class datasets by setting class 8 of the MNIST dataset to very low proportions of the other nine classes (e.g., 10:10:10:10:10:10:10:10:1:10 and 10:10:10:10:10:10:10:10:2:10). When class 8 is 0.1 fraction of the other nine classes, images of class 7 with a horizontal line outnumber images of class 8, and SLOGAN identifies 7 with a horizontal line as a more salient attribute than 8 as shown in the red boxes in Figure 21 (a). On the other hand, when class 8 is 0.2 fraction of the other nine classes, images of class 8 outnumber images of class 7 with a horizontal line. Therefore, SLOGAN successfully identifies 8 as a salient attribute as shown in the red box in Figure 21 (b).
Qualitative analysis with various imbalance ratios Figure 22 shows generated images from each latent component of SLOGAN trained on the AFHQ dataset. For various imbalance ratios of cats and dogs, we qualitatively analyze SLOGAN without using probe data. When the imbalance ratios are 1:1 and 1:2, SLOGAN identifies cat/dog as the most salient attribute and learned the attribute successfully as presented in Figure 22 (a) and (b). When the imbalance ratio is 1:5, SLOGAN discovers folded ears as the most salient attribute as shown in Figure 22 (c).

heading: Step 0 Step 1000 Step 5000 Step 10000Training data
text: CIFAR-2 (7:3) CIFAR-2 (9:1)
Figure 17: Generated images from each latent component of SLOGAN trained on the CIFAR-2 (7:3) and CIFAR-2 (9:1) datasets.
CD-GAN, PGMGAN, SLOGAN 이미지비교

heading: B ADDITIONAL BACKGROUND
text: Our work is closely related to Stein’s lemma and the reparameterization trick for Gaussian mixture and also related to several topics of GAN studies such as representation learning, supervised/unsupervised conditional generation, and Gaussian mixture prior.

heading: B.1 SUPERVISED CONDITIONAL GENERATION
text: Conditional GANs including ACGAN (Odena et al., 2017), projection discriminator (Miyato & Koyama, 2018), and ContraGAN (Kang & Park, 2020) have led to state-of-the-art performances in conditional image generation. However, these conditional GANs are trained with supervision and require a large amount of labeled data.

heading: B.2 UNSUPERVISED CONDITIONAL GENERATION
text: InfoGAN (Chen et al., 2016) introduces latent codes composed of categorical and continuous variables and derives the lower bound of mutual information between the latent codes and representations. By maximizing the lower bound, InfoGAN learns disentangled latent variables. ClusterGAN (Mukherjee et al., 2019) assumes a discrete-continuous Gaussian prior wherein discrete variables are defined as a one-hot vector and continuous variables are sampled from a Gaussian distribution. The latent distribution is clustered in an unsupervised manner through reconstruction loses for discrete and continuous variables. CD-GAN (Pan et al., 2021) is similar to ClusterGAN, but it uses contrastive loss to disentangle attributes. These methods assume uniform distribution of the attributes, and if the imbalance ratio is unknown, these models cannot thoroughly learn imbalanced attributes.
Recently, unsupervised conditional GANs which do not assume uniform distribution of the attributes have been proposed. Self-conditioned GAN (Liu et al., 2020) performs unsupervised conditional generation by using clustering of discriminator features as labels. However, it has no loss to facilitate clustering of discriminator features, and the performance seems to be largely influenced by the architecture of the discriminator. PGMGAN (Armandpour et al., 2021) applies a contrastive clustering method named SCAN (Van Gansbeke et al., 2020) to perform unsupervised conditional generation. The pretrained space partitioner and a penalization loss function are used to encourage the generator to generate images with salient attributes. To avoid degenerated clusters where one partition contains most of the data, PGMGAN uses an entropy term that regularizes the average cluster probability vector to a uniform distribution. However, when learning PGMGAN on datasets with imbalanced attributes, it is difficult to adjust the coefficient of the regularizer because the performance seems to be sensitive to the strength of the regularizer in practice (e.g., the results on CIFAR-2 (7:3) and (9:1) in Table 7). In addition, clustering results does not seem to be reliable when the space partitioner is trained on datasets where transformations are not available (e.g., the result on 10x 73k in Table 7).
NEMGAN (Mishra et al., 2020) argues that it considers the imbalance problem of attributes. However, it assumes that some labeled samples are provided, which is unrealistic in real-world scenarios. In addition, labeled samples should have the same imbalance ratio as the training data to estimate the imbalance ratio. If these samples are not given, NEMGAN is the same method as ClusterGAN. On the other hand, SLOGAN does not require labeled samples to learn imbalanced attributes. Even when a small amount of probe data is used to learn specific attributes, imbalanced attributes can be learned with balanced probe data (e.g., 30 male and 30 female faces on the CelebA-HQ dataset).

heading: B.3 REPRESENTATION LEARNING
text: Representation learning refers to the discovery of meaningful semantics in datasets (Bengio et al., 2013). Perceptually salient and semantically meaningful representations can induce better performance in downstream tasks. In particular, Representation learning in generative models enables the learned latent representations to be semantically meaningful without labels in the training data. BiGAN (Donahue et al., 2017), ALI (Dumoulin et al., 2017), and their variants (Jaiswal et al., 2018; Belghazi et al., 2018) are similar to our study in that they add an encoder network to the original GAN framework. The additional encoder may serve as a inversion mapping of the generator, and learned feature representation is useful for supervised tasks. However, the learned generator cannot perform conditional generation without supervision.

heading: C PROOFS
text: C.1 GRADIENT IDENTITY FOR µc
Theorem 1. Given an expected loss of the generator L and a loss function for a sample ℓ(·) : Rdz 7→ R, we assume ℓ is continuously differentiable. Then, the following identity holds:
∇µcL = Eq [δ(z)cπc∇zℓ(z)] (13)
Proof. To prove Theorem 1, the following lemma (Bonnet’s theorem) is introduced.
Lemma 2. Let h(z) : Rd 7→ R be continuously differentiable. q(z) is a multivariate Gaussian distribution N (z;µ,Σ). Then, the following identity holds:
∇µEq(z) [h(z)] = Eq(z) [∇zh(z)] (14)
The proof of Lemma 2 is described by Theorem 3 of Lin et al. (2019). Using Lemma 2, we show that
∇µcL = ∇µcEq(z) [ℓ(z)] = ∇µc K∑ i=1 p(i)Eq(z|i) [ℓ(z)] (15)
= p(c)∇µcEq(z|c) [ℓ(z)] = p(c)Eq(z|c) [∇zℓ(z)] (16)
= ∫ q(z|c)p(c)∇zℓ(z)dz (17)
= ∫ q(z)
q(z|c)p(c) q(z) ∇zℓ(z)dz (18)
= ∫ q(z)δ(z)cπc∇zℓ(z)dz (19)
= Eq(z) [δ(z)cπc∇zℓ(z)] (20)
C.2 FIRST-ORDER GRADIENT IDENTITY FOR Σc
Theorem 2. With the same assumptions from Theorem 1, we have the following gradient identity:
∇ΣcL = 1 2 Eq [ δ(z)cπcΣ −1 c (z− µc)∇Tz ℓ(z) ] (21)
Proof. In order to prove Theorem 2, we introduce the following lemma (Price’s theorem).
Lemma 3. Let h(z) : Rd 7→ R and its derivative∇h(z) be continuously differentiable. We further assume that E [h(z)] is well-defined. Then, the following identity holds:
∇ΣEq(z) [h(z)] = 1
2 Eq(z)
[ Σ−1(z− µ)∇Tz h(z) ] = 1
2 Eq(z)
[ ∇2zh(z) ] (22)
The proof of Lemma 3 is presented in Theorem 4 of Lin et al. (2019). The rest of the proof is similar with the proof of Theorem 1. Using the first-order gradient identity of Lemma 3, we get
∇ΣcL = ∇ΣcEq(z) [ℓ(z)] = ∇Σc K∑ i=1 p(i)Eq(z|i) [ℓ(z)] (23)
= p(c)∇ΣcEq(z|c) [ℓ(z)] = 1
2 p(c)Eq(z|c)
[ Σ−1c (z− µc)∇Tz ℓ(z) ] (24)
= 1
2
∫ q(z|c)p(c)Σ−1c (z− µc)∇Tz ℓ(z)dz (25)
= 1
2
∫ q(z)
q(z|c)p(c) q(z) Σ−1c (z− µc)∇Tz ℓ(z)dz (26)
= 1
2
∫ q(z)δ(z)cπcΣ −1 c (z− µc)∇Tz ℓ(z)dz (27)
= 1
2 Eq(z)
[ δ(z)cπcΣ −1 c (z− µc)∇Tz ℓ(z) ] (28)
C.3 ENSURING POSITIVE-DEFINITENESS OF Σc
Theorem 3. The updated covariance matrix Σ′c = Σc+γ∆Σ′c with the modified update rule specified in Equation 6 is positive-definite if Σc is positive-definite.
Proof. Because Σc is symmetric and positive-definite, we can decompose Σc = LLT using Cholesky decomposition, where L is the lower triangular matrix. Then, we can prove the positive-definiteness of the updated covariance matrix as follows:
Σ′c = Σc + γ∆Σ ′ c (29)
= Σc + γ(∆Σc + γ
2 ∆ΣcΣ
−1 c ∆Σc) (30)
= Σc + γ∆Σc + γ2
2 ∆ΣcΣ
−1 c ∆Σc (31)
= 1
2
( Σc + (L + γ∆ΣcL −T )(LT + γL−1∆Σc) )
(32)
(33)
Let us define U := LT + γL−1∆Σc. Then, we have the following:
Σ′c = 1
2
( Σc +U TU ) ≻ 0 (34)
where both Σc and UTU are positive-definite, concluding the proof.
C.4 GRADIENT IDENTITY FOR ρc
Theorem 4. Let ρc be a mixing coefficient parameter, and the following gradient identity holds:
∇ρcL = Eq [πc (δ(z)c − 1) ℓ(z)] (35)
Proof. The gradient of the latent distribution with respect to the mixing coefficient parameter is derived as follows:
∇ρcq(z) = ∇ρc K∑ i=1 softmax(ρi)q(z|i) = πc
( q(z|c)−
K∑ i=1 πiq(z|i)
) (36)
where softmax(·) denotes the softmax function (e.g., p(i) = πi = softmax(ρi)). Using the above equation, we have
∇ρcL = ∇ρcEq(z) [ℓ(z)] = ∫ ∇ρcq(z)ℓ(z)dz (37)
= ∫ πc ( q(z|c)−
K∑ i=1 πiq(z|i)
) ℓ(z)dz (38)
= ∫ πcq(z) ( q(z|c) q(z) − K∑ i=1 πi q(z|i) q(z) ) ℓ(z)dz (39)
= ∫ q(z)πc ( δ(z)c −
K∑ i=1 πiδ(z)i
) ℓ(z)dz (40)
Here, ∑K i=1 πiδ(z)i = ∑K i=1 p(i)q(z|i) q(z) = 1. Plugging this back into Equation 40, we obtain
∇ρcL = ∫ q(z)πc (δ(z)c − 1) ℓ(z)dz (41)
= Eq(z) [πc (δ(z)c − 1) ℓ(z)] (42)
C.5 SECOND-ORDER GRADIENT IDENTITY FOR Σc
We consider a generator σ(G(z)) = σ(G) = σG, a discriminator D(σG) = D with the piecewiselinear activation functions (e.g. ReLU or LeakyReLU) except the activation function of the output layer of the generator which is the hyperbolic tangent function σ(x) = (ex− e−x)/(ex+ e−x), and a Wasserstein adversarial loss function for each latent vector ℓ(z) = −D(σ(G(z))) = ℓ. We note that the following holds except on a set of zero Lebesgue measure because the piecewise-linear activation functions are linear except where they switch:
∂2D
∂σG2 = 0,
∂2G ∂z2 = 0 (43)
We provide the second-order gradient identity of the Wasserstein GAN loss for the covariance matrix Σc that does not compute second-order derivatives. However, it is impractical and not included in our method because of the excessive computational cost of Jacobian matrices. Theorem 5. Given the Wasserstein GAN loss for the generator L and a loss function for a sample ℓ(z) : Rd 7→ R, we assume ℓ and its derivative ∇ℓ(z) are continuously differentiable. We further assume that E [ℓ(z)] is well-defined. Then, the following identity holds:
∇ΣcL = ∇ΣcEq(z) [ℓ(z)] = −Eq(z) [ πcδ(z)c∇Tz G(z) diag ( ∇σGD ⊙ (σ◦3G − σG) ) ∇zG(z) ] where ⊙ denotes element-wise multiplication of vectors and x◦i denotes the i-th Hadamard (elementwise) power of a vector x.
Proof. From the second-order identity of Lemma 3, we have the following:
∇ΣcL = ∇ΣcEq(z) [ℓ(z)] = ∇Σc K∑ i=1 p(i)Eq(z|i) [ℓ(z)] (44)
= p(c)∇ΣcEq(z|c) [ℓ(z)] = 1
2 p(c)Eq(z|c)
[ ∇2zℓ(z) ] (45)
= 1
2 Eq(z)
[ q(z|c)p(c)
q(z) ∇2zℓ(z)
] (46)
= 1
2 Eq(z)
[ πcδ(z)c∇2zℓ(z) ] (47)
The Hessian of the sample loss with respect to z is given as follows: ∇2zℓ(z) = ∇z ( −∇Tz D ) = ∇z ( −(∇GσG∇zG)T∇σGD )T (48)
= ∇z ( −(∇σGD)T∇GσG∇zG ) (49)
= − ∂ ∂z
[ ∂D
∂(σG)1 (σ′G)1 . . . ∂D ∂(σG)dx (σ′G)dx ] ∂G1 ∂z1 . . . ∂G1∂zdz ... . . . ...
∂Gdx ∂z1 . . . ∂Gdx ∂zdz  (50) = − ∂
∂z
[∑dx i=1 ∂D ∂(σG)i (σ′G)i ∂Gi ∂z1 . . . ∑dx i=1 ∂D ∂(σG)i (σ′G)i ∂Gi ∂zdz ] (51)
= −  ∂ ∂z1 (∑dx i=1 ∂D ∂(σG)i (σ′G)i ∂Gi ∂z1 ) . . . ∂∂z1 (∑dx i=1 ∂D ∂(σG)i (σ′G)i ∂Gi ∂zdz ) ... . . . ...
∂ ∂zdz (∑dx i=1 ∂D ∂(σG)i (σ′G)i ∂Gi ∂z1 ) . . . ∂∂zdz (∑dx i=1 ∂D ∂(σG)i (σ′G)i ∂Gi ∂zdz
)  (52)
where σ′G = ∇GσG. To simplify an element (∇2zℓ(z))jk = − ∂∂zj (∑dx i=1 ∂D ∂(σG)i (σ′G)i ∂Gi ∂zk ) for arbitrary j, k ∈ {1, ..., dz}, we have
∂2D
∂zj∂(σG)i =
∂
∂(σG)i
∂D ∂zj =
∂
∂(σG)i ( dx∑ l=1 ∂D ∂(σG)l ∂(σG)l ∂zj ) (53)
= ∂
∂(σG)i ( dx∑ l=1 ∂D ∂(σG)l (σ′G)l ∂Gl ∂zj ) (54)
= dx∑ l=1
∂ 2D
∂(σG)i∂(σG)l (σ′G)l ∂Gl ∂zj + dx∑ l=1 ∂D ∂(σG)l ∂(σ′G)l ∂(σG)i ∂Gl ∂zj
(55)
+ dx∑ l=1 ∂D ∂(σG)l (σ′G)l ∂2Gl ∂(σG)i∂zj
(56)
= ∂D
∂(σG)i (−2(σG)i) ∂Gi ∂zj + ∂D ∂(σG)i (σ′G)i ∂ ∂zj
( 1
(σ′G)i
) (57)
= ∂D
∂(σG)i (−2(σG)i) ∂Gi ∂zj + ∂D ∂(σG)i (σ′G)i 2(σG)i (σ′G)i ∂Gi ∂zj
(58)
= 0 (59)
dx∑ i=1 ∂D ∂(σG)i ∂(σ′G)i ∂zj ∂Gi ∂zk = dx∑ i=1 ∂D ∂(σG)i ∂(σ′G)i ∂Gi ∂Gi ∂zj ∂Gi ∂zk
(60)
= dx∑ i=1 ∂D ∂(σG)i ( 2(σG) 3 i − 2(σG)i ) ∂Gi ∂zj ∂Gi ∂zk
(61)
dx∑ i=1 ∂D ∂(σG)i (σ′G)i ∂ 2Gi ∂zj∂zk = 0 (62)
Therefore, the simplified element is
(∇2zℓ(z))jk = − dx∑ i=1 ∂D ∂(σG)i ( 2(σG) 3 i − 2(σG)i ) ∂Gi ∂zj ∂Gi ∂zk
(63)
We now vectorize the expression of∇2zℓ(z) as follows: ∇2zℓ(z) = −2 ( ∂G
∂z
)T diag ( ∂D
∂σG
) diag ( σ◦3G − σG )(∂G ∂z ) (64)
Plugging this to Equation 47, the following is obtained: ∇ΣcL = −Eq(z) [ πcδ(z)c∇Tz Gdiag ( ∇σGD ⊙ (σ◦3G − σG) ) ∇zG) ] (65)

heading: D METHODOLOGICAL DETAILS

heading: D.1 ADDITIVE ANGULAR MARGIN
text: To enhance the discriminative power of U2C loss, we adopted the additive angular margin (Deng et al., 2019) as follows:
ℓU2C(z i) = − log exp(s · cos(θii +m))1 B {exp(s · cos(θii +m)) + ∑ j ̸=i exp(s · cos θij)} (66)
where s denotes the feature scale, and m is the angular margin. The feature scale m and the coefficient of U2C loss λ are linearly decayed to 0 during training, so that SLOGAN can focus more on the adversarial loss as training progresses.

heading: D.2 CLUSTER ASSIGNMENT
text: In Section 3.3, we chose cos θic as the critic function assuming that it is proportional to log p(c|xg). If the real data distribution p(xr) and the generator distribution p(xg) are sufficiently similar via adversarial learning, the cosine similarity between E(xr) and µc can also be considered proportional to log p(c|xr). Therefore, for real data, we obtain the probability for each cluster as follows:
p̂(c|xr) = exp(cos θc)∑K
k=1 exp(cos θk) (67)
where cos θk = E(xr) ·µk/∥E(xr)∥∥µk∥ is the cosine similarity between E(xr) and µk. The data can then be assigned to the cluster with the highest probability (i.e., argmaxc p̂(c|xr)).

heading: D.3 ATTRIBUTE MANIPULATION
text: We utilized mixup (Zhang et al., 2018) to make the best use of a small amount of probe data when manipulating attributes. Algorithm 3 describes the procedure for using mixup for attribute manipulation when K = 2. We applied the same feature scale and angular margin to Lm, as shown in Equation 66. The number of iterations for the mixup (T ) was set to five.
Algorithm 3 Attribute manipulation
Initialize probe data with the desired attribute Xc=1 ← {xic=1}Mi=1 and X̄c=1 ← {xic=1}Mi=1 Initialize probe data without the desired attribute Xc=0 ← {xic=0}Mi=1 and X̄c=0 ← {xic=0}Mi=1 for each mixup iteration t in {1, ..., T} do X̄c=1 ← X̄c=1 ∪MIXUP(Xc=1, PERMUTE(Xc=1)) X̄c=0 ← X̄c=0 ∪MIXUP(Xc=0, PERMUTE(Xc=0)) end for for each augmented data index j in {1, ...,M(T + 1)} do cos θj00 ← E(x̄ j c=0) · µ0/∥E(x̄ j c=0)∥∥µ0∥
cos θj01 ← E(x̄ j c=0) · µ1/∥E(x̄ j c=0)∥∥µ1∥ cos θj10 ← E(x̄ j c=1) · µ0/∥E(x̄ j c=1)∥∥µ0∥ cos θj11 ← E(x̄ j c=1) · µ1/∥E(x̄ j c=1)∥∥µ1∥
end for Lm = − 1M(T+1) ∑M(T+1) j=1 log
exp(s·cos(θj00+m))+exp(s·cos(θ j 11+m))
exp(s·cos(θj00+m))+exp(s·cos θ j 01)+exp(s·cos θ j 10)+exp(s·cos(θ j 11+m))
Minimize Lp with respect to E, G, and µ

heading: D.4 SIMCLR
text: For the CIFAR and CelebA datasets, we used the SimCLR loss (Chen et al., 2020) for the encoder. We applied color, translation, and cutout transformations to the generated data using DiffAugment1
1https://github.com/mit-han-lab/data-efficient-gans
(Zhao et al., 2020). The SimCLR loss is calculated using the generated data xig and augmentation A as follows:
ℓSimCLR(z i) = − log exp(E(xig) · E(A(xig)) / ∥E(xig)∥∥E(A(xig))∥)∑B j=1 exp(E(x i g) · E(A(x j g)) / ∥E(xig)∥∥E(A(x j g))∥)
(68)
where xig = G(z i). The encoder E is trained to minimize 1B ∑B i=1 ( ℓadv(z i) + ℓSimCLR(z i)
+λℓU2C(z i) ) .

heading: D.5 DELIGAN+
text: Among the existing unsupervised conditional GANs, DeLiGAN lacks an encoder network. Therefore, for a fair comparison, we added an encoder network and named it DeLiGAN+. We set the output dimension of the encoder to equal the number of mixture components of the latent distribution. For the i-th example in the batch, when the ci-th mixture component of the latent distribution is selected, DeLiGAN+ is learned through the following objective:
min G,E,µc,σc max D
1
B B∑ i=1 [ D(xi)−D(G(zi, ci))− λCE cTi logE(G(zi, ci)) ] (69)
where ci is the one-hot vector corresponding to ci and λCE is the coefficient of the cross entropy loss. We set λCE to 10 in the experiments.

heading: D.6 EVALUATION METRIC
text: Cluster assignment We do not use clustering purity which is an evaluation metric for cluster assignment. To compute the clustering purity, the most frequent class in the cluster is obtained, and the ratio of the data points belonging to the class is calculated. However, if the attributes in the data are imbalanced, multiple clusters can be assigned to a single class in duplicate, and this high clustering purity misleads the results. Therefore, we utilized the normalized mutual information (NMI) implemented in scikit-learn2.
Unconditional generation FID has the advantage of considering not only sample quality but also diversity, whereas Inception score (IS) cannot assess the diversity properly because IS does not compare generated samples with real samples (Shmelkov et al., 2018). Therefore, we used FID as the evaluation metric for unsupervised generation.
Unsupervised conditional generation If attributes in data are severely imbalanced, FID does not increase (deteriorate) considerably even if the model does not generate data containing the minority attributes. Therefore, the FID cannot accurately measure the unsupervised conditional generation performance for data with severely imbalanced attributes. We introduce ICFID to evaluate the performance of unsupervised conditional generation. When calculating ICFID, multiple clusters cannot be assigned to a single class in duplicate. Therefore, if data of a single class are generated from multiple discrete latent variables or modes, the model shows high (bad) ICFID.
2https://github.com/scikit-learn/scikit-learn/blob/15a949460/sklearn/ metrics/cluster/_supervised.py
E IMPLEMENTATION DETAILS

heading: E.1 GENERAL SETTINGS AND ENVIRONMENTS
text: For simplicity, we denote the learning rate of G as η and the learning rate of Σ as γ. Throughout the experiments, we set the learning rate of E to η, and D to 4η using the two-timescale update rule (TTUR) (Heusel et al., 2017). We set the learning rate of µ to 10γ, and the learning rate of ρ to γ. We set B to 64 and the number of training steps to 100k. To stabilize discriminator learning, we used Lipschitz penalty (Petzka et al., 2018) for the synthetic, MNIST, FMNIST, and 10x 73k datasets, and adversarial Lipschitz regularization (Terjék, 2020) for the CIFAR-10 and CelebA datasets. We repeated each experiment 3 times and reported the means and standard deviations of model performances. Hyperparameters are determined by a grid search. We used the Adam optimizer (Kingma & Ba, 2014) for training D, G, and E, and a gradient descent optimizer for training µ, Σ, and ρ. The experiments herein were conducted on a machine equipped with an Intel Xeon Gold 6242 CPU and an NVIDIA Quadro RTX 8000 GPU. The code is implemented in Python 3.7 and Tensorflow 1.14 (Abadi et al., 2016).

heading: E.2 SYNTHETIC DATASET
text: For the synthetic dataset, we first set the mean of eight 2-dimensional Gaussian distributions as (0, 2), ( √ 2, √ 2), (2, 0), ( √ 2,− √ 2), (0,−2), (− √ 2,− √ 2), (−2, 0), and (− √ 2, √ 2), and the variance as 0.01I . In Figure 11, we also set the variances as 0.05I and 0.1I . The number of data sampled from the Gaussian distributions was set to 5,000, 5,000, 5,000, 5,000, 15,000, 15,000, 15,000, and 15,000. We scaled a total of 80,000 data points to a range between -1 and 1. Table 11 shows the network architectures of SLOGAN used for the synthetic dataset. Linear n denotes a fully-connected layer with n output units. BN and SN denote batch normalization and spectral normalization, respectively. LReLU denotes the leaky ReLU. We set λ = 4, η = 0.001, γ = 0.01, s = 2, and m = 0.5.

heading: E.3 MNIST AND FASHION-MNIST DATASETS
text: The MNIST dataset (LeCun et al., 1998) consists of handwritten digits, and the Fashion-MNIST (FMNIST) dataset (Xiao et al., 2017) is comprised of fashion products. Both the MNIST and FashionMNIST datasets have 60,000 training and 10,000 test 28×28 grayscale images. Each pixel was scaled to a range of 0−1. The datasets consist of 10 classes, and the number of data points per class is balanced. Table 12 shows the network architectures of SLOGAN used for the MNIST and FMNIST datasets. Conv k × k, s, n denotes a convolutional network with n feature maps, filter size k × k, and stride s. Deconv k × k, s, n denotes a deconvolutional network with n feature maps, filter size k × k, and stride s. For the MNIST dataset, we set λ = 10, η = 0.0001, γ = 0.002, s = 8, and m = 0.5. For MNIST-2, we set λ = 4, η = 0.0001, γ = 0.002, s = 4, and m = 0.5. For the FMNIST dataset, we set λ = 10, η = 0.0001, γ = 0.001, s = 1, and m = 0. For FMNIST-5, we set λ = 1, η = 0.0002, γ = 0.004, s = 4, and m = 0.5.
E.4 10X 73K DATASET
The 10x 73k dataset (Zheng et al., 2017) consists of 73,233 720-dimensional vectors, which are obtained from RNA transcript counts, and has eight cell types (classes). The number of data points per cell type is 10,085, 2,612, 9,232, 8,385, 10,224, 11,953, 10,479, and 10,263. We converted each element to logscale (i.e., log2(x + 1)) and scaled each element to a range between 0 and 1. Table 13 shows the network architectures of SLOGAN used for the 10x 73k dataset. We set λ = 10, η = 0.0001, γ = 0.004, s = 4, and m = 0.

heading: E.5 CIFAR-10 DATASET
text: The CIFAR-10 (Krizhevsky et al., 2009) dataset comprises 50,000 training and 10,000 test 32×32 color images. Each pixel was scaled to a range of -1 to 1. The number of data points per class is balanced. Figure 23 and Table 14 show the network architectures of residual blocks and SLOGAN used for the CIFAR datasets. AvgPool and GlobalAvgPool denote the average pooling and global average pooling layers, respectively. For the CIFAR-10, CIFAR-2, CIFAR-2 (7:3), and CIFAR-2 (9:1) datasets, we set λ = 1, η = 0.0001, γ = 0.002, s = 4, and m = 0.5.

heading: E.6 CELEBA DATASET
text: The CelebA dataset (Liu et al., 2015) consists of 202,599 face attributes. We cropped the face part of each image to 140×140 pixels, resized it to 64×64 pixels, and scaled it to a range between -1 and 1.
The imbalanced ratio is different for each attribute, and the imbalanced ratios of male and eyeglasses used in the experiment are 1:1 and 14:1, respectively. Table 15 lists the network architectures of SLOGAN used for the CelebA dataset. We set λ = 1, η = 0.0002, γ = 0.0006, s = 4, and m = 0.5.

heading: E.7 CELEBA-HQ DATASET
text: The CelebA-HQ dataset (Karras et al., 2017) consists of 30,000 face attributes. We resized each image to 128×128 and 256×256 pixels, and scaled it to a range between -1 and 1. The imbalance ratio of male used in the experiment was 1.7:1. We used StyleGAN2 (Karras et al., 2020) architecture with DiffAugment3 and applied implicit reparameterization to the input space of the mapping network. We set λ = 1, η = 0.002, γ = 0.0006, s = 1, and m = 0.

heading: E.8 AFHQ DATASET
text: The AFHQ dataset (Choi et al., 2020) consists of 15,000 high-quality animal faces. We used cats and dogs, and there are about 5,000 images each in the dataset. We resized each image to 256×256 pixels, and scaled it to a range between -1 and 1. We set the imbalance ratios of cats and dogs to 1:1, 1:2, and 1:5 by reducing the number of cats in the training dataset. We used the same model architecture and hyperparameters as for the CelebA-HQ dataset.

heading: E.9 CODE AVAILABILITY
text: Code is available at https://github.com/shinyflight/SLOGAN
3https://github.com/mit-han-lab/data-efficient-gans/tree/master/ DiffAugment-stylegan2
REFERENCES IN APPENDIX Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin,
Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp. 265–283, 2016.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214–223. PMLR, 2017.
Mohammadreza Armandpour, Ali Sadeghian, Chunyuan Li, and Mingyuan Zhou. Partition-guided gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5099–5109, 2021.
Mohamed Ishmael Belghazi, Sai Rajeswar, Olivier Mastropietro, Negar Rostamzadeh, Jovana Mitrovic, and Aaron Courville. Hierarchical adversarially learned inference. arXiv preprint arXiv:1802.01071, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PMLR, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In Neural Information Processing Systems (NIPS), 2016.
Jacob Cohen. Statistical power analysis for the behavioral sciences. Routledge, 2013.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690–4699, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In 5th International Conference on Learning Representations, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martı́n Arjovsky, Olivier Mastropietro, and Aaron C. Courville. Adversarially learned inference. In 5th International Conference on Learning Representations, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6629–6640, 2017.
Ayush Jaiswal, Wael AbdAlmageed, Yue Wu, and Premkumar Natarajan. Bidirectional conditional generative adversarial networks. In Asian Conference on Computer Vision, pp. 216–232. Springer, 2018.
Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. In Advances in Neural Information Processing Systems, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
Wu Lin, Mohammad Emtiyaz Khan, and Mark Schmidt. Stein’s lemma for the reparameterization trick with exponential family mixtures. arXiv preprint arXiv:1910.13398, 2019.
Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, and Antonio Torralba. Diverse image generation via self-conditioned gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14286–14295, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Deepak Mishra, Aravind Jayendran, and AP Prathosh. Effect of the latent structure on clustering with gans. IEEE Signal Processing Letters, 27:900–904, 2020.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In International Conference on Learning Representations, 2018.
Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan. Clustergan: Latent space clustering in generative adversarial networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4610–4617, 2019.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning, pp. 2642–2651. PMLR, 2017.
Lili Pan, Peijun Tang, Zhiyong Chen, and Zenglin Xu. Contrastive disentanglement in generative adversarial networks. arXiv preprint arXiv:2103.03636, 2021.
Henning Petzka, Asja Fischer, and Denis Lukovnikov. On the regularization of wasserstein GANs. In International Conference on Learning Representations, 2018.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In Proceedings of the European Conference on Computer Vision (ECCV), pp. 213–229, 2018.
Dávid Terjék. Adversarial lipschitz regularization. In International Conference on Learning Representations, 2020.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European Conference on Computer Vision, pp. 268–285. Springer, 2020.
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics, pp. 196–202. Springer, 1992.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33, 2020.
Grace XY Zheng, Jessica M Terry, Phillip Belgrader, Paul Ryvkin, Zachary W Bent, Ryan Wilson, Solongo B Ziraldo, Tobias D Wheeler, Geoff P McDermott, Junjie Zhu, et al. Massively parallel digital transcriptional profiling of single cells. Nature communications, 8(1):1–12, 2017.

Review summaries:
This paper uses Gaussian mixture as a prior to address the scenario of imbalanced attributes in GANs latent space clustering. The study derives Stein latent optimization that provides reparameterizable gradient estimations when assuming a Gaussian mixture prior for the latent space. The results show that the proposed model achieves superior performance compared to the baselines. 

The paper is concerned with unsupervised image generation using GANs. They devise a solution (SLOGAN) for addressing the problems current models have, especially when dealing with datasets with unbalanced features/attributes. In addition, to learn attributes from data and improve the conditional generation, it suggests using a new contrastive loss (U2C).   

This work proposes a method for performing unsupervised conditional generation from imbalanced attributes. The key components of this work are the following:  * A (architecture agnostic) GAN with a GMM latent space, where we have one component per attribute in the data.   * The parameters of the components $(\mu, \Sigma)$ are learned via implicit reparameterization, and leveraging stein's lemma to derive gradients for $\mu$ and $\Sigma$.   * The components are encouraged to be distinct by introducing a contrastive objective. An encoder is used to obtain a low dimensional vector representation of a generated sample, and this sample is encouraged to be close to the mean vector of the corresponding component in the latent space. 

The paper proposed a Stein Latent Optimization for Generative adversarial networks, which can perform conditional generation in an unsupervised manner. The core innovation is the reparameterizable gradient estimations. The proposed unsupervised conditional contrastive loss further ensures the single attribute generation capacity. The idea is somewhat novel and motivated. The authors perform empirical studies on diverse datasets to conclude that the proposed algorithm is effective in learning balanced or imbalanced attributes.