Information-theoretic stochastic contrastive conditional GAN: InfoSCC-GAN
Authors: Vitaliy Kinakh
Mariia Drozdova
Guillaume Quétant
Svyatoslav Voloshynovskyy
Tobias GOLLING
Venue: ICLR 2022 Submitted (2022)
Keywords: GANs
Generative adversarial networks
Contrastive learning
Conditional image generation

Abstract: Conditional generation is a subclass of generative problems when the output of generation is conditioned by a class attributes’ information. In this paper, we present a new stochastic contrastive conditional generative adversarial network (InfoSCC-GAN) with explorable latent space. The InfoSCC-GAN architecture is based on an unsupervised contrastive encoder built on the InfoNCE paradigm, attributes' classifier, and stochastic EigenGAN generator. 
We propose two approaches for selecting the class attributes: external attributes from the dataset annotations and internal attributes from the clustered latent space of the encoder. We propose a novel training method based on a generator regularization using external or internal attributes every $n$-th iteration using the pre-trained contrastive encoder and pre-trained attributes’ classifier.  The proposed InfoSCC-GAN is derived from an information-theoretic formulation of mutual information maximization between the input data and latent space representation for the encoder and the latent space and generated data for the decoder. Thus, we demonstrate a link between the training objective functions and the above information-theoretic formulation. The experimental results show that InfoSCC-GAN outperforms vanilla EigenGAN in image generation on several popular datasets, yet providing an interpretable latent space. In addition, we investigate the impact of regularization techniques and each part of the system by performing an ablation study. Finally, we demonstrate that thanks to the stochastic EigenGAN generator, the proposed framework enjoys a truly stochastic generation in contrast to vanilla deterministic GANs yet with the independent training of an encoder, a classifier, and a generator. 
The code, supplementary materials, and demos are available \url{https://anonymous.4open.science/r/InfoSCC-GAN-D113}

Under review as a conference paper at ICLR 2022
C INFORMATION-THEORETI STOCHASTIC
CONTRASTIVE CONDITIONAL GAN: INFOSCC-GAN
Anonymous authors Paper under double-blind review
ABSTRACT
Conditional generation is a subclass of generative problems when the output of generation is conditioned by a class attributes' information. In this paper, we present a new stochastic contrastive conditional generative adversarial network (InfoSCC-GAN) with explorable latent space. The InfoSCC- GAN architecture is based on an unsupervised contrastive encoder built on the InfoNCE paradigm, attributes" classifier, and stochastic EigenGAN generator. We propose two ap- proaches for selecting the class attributes: external attributes from the dataset an- notations and internal attributes from the clustered latent space the encoder. We propose novel training method based on a generator regularization using ex- ternal or internal attributes every n-th iteration using the pre-trained contrastive encoder and pre-trained attributes' classifier. The proposed InfoSCC-GAN is de- formulation mutual information maximiza-
rived from an information-theoreet tion between the input data and latent space representation for the encoder and the latent space and generated data for the decoder. Thus, we demonstrate a link between the training objective functions and the above information -tteorrtiict for- mulation. The experimental results show that InfoSCC-GAN outperforms vanilla EigenGAN in image generation on several popular datasets, yet providing an interpretable latent space. In addition, we investigate the impact of regulariza- tion techniques and each part of the system by performing an ablation study. Finally, we demonstrate that thanks to the stochastic EigenGAN generator, the proposed framework enjoys a truly stochastic generation in contrast to vanilla de- terministic GANs yet with the independent training of an encoder, classifier, and a generator. The code, supplementary materials, and demos are available https::/ htttp:://anonymouss 4oper. wwtsecc/re/eenoeeeeeee/cc// GAN DAN-D113
INTRODUCTION
Conditional image generation is the task of generating images, based some attributes. The idea of conditional GAN (cGAN) was proposed by Mirza & Osindero (2014). The authors modified the classic GAN architecture by adding the attribute
parameter to the input of the generator to generate the corresponding image. They also added attributes to the discriminator input to better distinguish real data. Since then, lot of other methods have been developed. ACGAN (Odena et al. (2017)) has an auxiliary classifier to guide the generator to synthesize well-classifiable images. approach proposed in ACGAN by utilizing the inner product of an embedded image and the corresponding attribute embeddings. ContraGAN (Kang Park (2200) utilizes contrastive 2C' loss with multiple positive and negative pairs to update the generator.
ProjGAN (Miyato & Koyama (2018) improves
While these methods have shown impressive results in conditional image generation, they are known to be difficult to train, to lack the image diversity within the same input attribute, and not to always have meaningful data exploration the latent space.
We propose new stochastic contrastive conditional generative adversarial network (InfoSCC-GAN) with an explorable latent space. As a baseline generator, we use EigenGAN (He et al. (202)) gen- erator with interpretable and controllable input dimensions, yet trained in unsupervised way. EigenGAN ensures that different layers of generative CNN, controlled by noise vectors, hold dif- ferent semantics of the synthesized images. EigenGAN is able to mine interpretable and controllable

---

Under review as a conference paper at ICLR 2022
dimensions in unsupervised way from different generator layers by embedding one linear sub- space with an orthogonal basis into each generator layer. These layer-wise subspaces automatically discover set of "figff-eieedddd "igenndimenssons" at each layer corresponding to set of semantic attributes or in- terpretable variations, via generative adversarial training to learn a target distribution. By traversing the coefficient of specific "eigen-dimension"" the generator can generate samples with continuous changes corresponding to specific semantic attribute. The "core" latent space of EigenGAN is gen- erated from the Gaussian distribution. In contrast that, we use the latent space of the contrastive encoder as the "core" latent space for the generator. By using the contrastive encoder, we have the ability discover the "inner" attributes from the dataset by clustering the latent space. The "inner" attributes are useful for datasets without external annotations, unbalanced datasets, and datasets with subclasses. By using the encoder, we have an opportunity to compare latent spaces of the real im- ages and the generated images. At the same time, the classifier ensures the correspondence between the attributes of training data and conditionally generated ones. Also, by training the encoder and the classifier independently from the generator, reduce the general training complexity of the system. It allows avoiding training the encoder and classifier on unrealistic synthetic data, when training it jointly with the generator and discriminator.
The information-theoret experiments and ablation studies are provided in Section
interpretation of the proposed model is provided in Section The
We summarize our contributions as follows:
We proposed novel Stochastic Contrastive Conditional Generative Adversarial Network (InfoSCC-GAN for stochastic conditional image generation with controllable and inter- pretable latent space. is based on an EigenGAN, an independent contrastive encoder, and independent attribute classifier.
We introduce a novel classification regularization technique, which is based on updating the model with classification loss each n-th iteration and updating the generator using the adversarial and classification loss separately.
We propose novel method for the attribute selection, based clustering the embeddings, computed using the pre-trained contrastive encoder.
We provide an information- interpretation of the proposed system.
We perform an ablation study to determine the contribution of each part of the model to overall performance.
INFORMATION--THOREEICCA FORMULATION
2.1 THE TRAINING OF THE ENCODER (STAGE 1)
The encoder training is schematically shown in Figure stage 1. The encoder training is based on the maximization problem:
He argmax Ts. (X:E),
where (X; Epix.e) log (x(x) (6)
where qo. (e(x) denotes the encoder and qo, (E) the
marginal latent space distribution.
In the framework of contrastive learning, 1 maximized based on the infoNCE framework van den Oord et al. (2018)). In the practical implementation, one can use approaches similar to SimCLR (Chen et al. (2020) where the inner product between the positive pairs created from the augmented views originating from the same image is maximized and the inner product between the negative pairs originating from different images is minimized. Alternatively, one can use other ap- proaches to learn the representation such as BYOL (Grill et al. (2020)) Barlow Twins (Zbontar et al. (202T)). etc. without loose the generality of the proposed approach. It should be pointed out that the encoder is trained independently from the decoder in the scope of the considered setup.
2

---

Under review as a conference paper at ICLR 2022
Stage
Stage 2
Py(y)
(x) Me. (=(x)
Ur, (hjs) -h
Ci b/)
pxx,y
4,(x,8)
05, (c/x)
P. c
(00()) No (elx)) - Wes. (hie)
(x(x)
Stage
Dar(X
(y
Fele)
Gyly,9)
16. (c/xc)
[yle)
(x)
66. (F)
Pe, (y)
Figure I: Overview of the proposed InfoSCC-GAN training. Stage 1. Contrastive encoder training. Stage Classifier training. Stage 3. Conditional generator training.
2.2 THE TRAINING OF THE CLASS ATTRIBUTE CLASSIFIER (STAGE 2)
The class attribute classifier training is schematically shown in Figure 1 stage 2. The training of the class attribute classifier is based on the maximization problem:
0y argmax Tøz.0, (Y:E),
By
E) and H(Y) (xy) log log (y) and the conditional entropy is defined as Høzee o, EY - Exx Ea (e(x) [log Pe, (y [)] Since H(Y) is inde-
where 10:.0, (Y:E) H(Y) Hor e, EY
pendent of the parameters of the encoder and classifier, (2) reduces to the lower bound minimization:
0, argmin Hor.o, (Y(E),
6,
that under the categorical conditional distribution pe, (yle) can be expressed the categorical cross- entropy Ly(x g))
2.3
THE TRAINING OF THE DECODER, I.E., THE MAPPER AND GENERATOR (STAGE 3)
The training of decoder is shown in Figure [ stage 3. The decoder trained first to maximize the mutual information between the class attributes y predicted from the generated images and true class attributes
(0x, 2b) arggaax I 0.0..00 02 Gz.
(Y; (Y;
(4)
where A..0..0.0 (Y; E)
H(Y) Hv.0. (Y and H(Y)
-E -Epu(y) log logpy(y) and the conditional entropy is defined as He 0; (YE
where pø, (y responds to the classifier and 43, (e|x) denotes the pre-trained encoder. Since H(Y) is independent of the parameters of the encoder and classifier, (4) reduces to the lower bound minimization:
-Eps(y) (pp(() (xp(ellve)) (xlee (eg (log)o
e)
(0 2b) argmin H. 0z. (Y
(5)

---

Under review as a conference paper at ICLR 2022
that under the categorical conditional distribution po, (yle) can be expressed the categorical cross- entropy (y, y).
Finally, the decoder should produce samples that follow the distribution of training data px(x) that corresponds to the maximization of mutual information:
(0x, +b)
argmax IN E); 0x
(6)
where Iv.0. (X;
(x) Bxy(y) Epa(2) E Brylesy E Bre, (xis)
log Pas (x)e)
px(x)
(xy( [(p))()) (ely..x) (pe, (x[E e)llpo, (x))]]] - DeL. (xx
po. (x)), where pe_ (x)
denotes the distribution of generated samples x. Since (po, (x|E E) Po. (x)) 2 0, the maxi-
mization of the above mutual information reduces to the minimization problem:
(0z. 1)) argmin D (xx (x)). 0..0
The above discriminator is denoted as xxx(x), At the same time, one can also envision the discrim- inator conditioned on the attribute class y Dxx(x y) that is implemented as set of discriminators for each subset of generated and original samples defined by
3 IMPLEMENTATION DETAILS
Dataset We test the proposed method on AFHQ (Choi et al. (2020) and CelebA (Liu et al. (2015)) datasets. AFHQ dataset contains 16130 images belonging to classes: cats, dogs, and wilds animals, CelebA dataset contains 202599 face images with 40 binary attributes. We use AFHQ and CelebA dataset for visual result inspection and AFHQ dataset for the ablations studies.
3.1
ENCODER
The proposed encoder designed to produce the interpretable latent space and can be used for: (i) internal latent exploration, (ii) feature metric like the VGG-loss (Ledig et al. (2017)), (iii) feature extraction for the classification of the generated samples with "external" and "internal" labels.
We have selected the SimCLR unsupervised encoder since it has shown -fate-of-the-arr performance in unsupervised learning on diverse datasets. By training the encoder in an unsupervised way, is trained to learn the inner data distribution, which is then used compare real and generated data. For both AFHQ and CelebA datasets we use Resnet50 (He et al. (2016) as a base model. pretrain the SimCLR model for each dataset using contrastive NT-Xent loss (Sohn (2016)) We apply the same augmentations as in the original SimCLR paper. The 2D t-SNE of the extracted features for the AFHQ dataset is shown in Figure 2 The 2D t-SNEs of the extracted features for CelebA dataset for selected attributes are shown in the Appendix
3.2 CLASSIFIER
The initial idea of using a pre-trained classifier to regularize the generative model is based on the need to generate images that belong to the specific class. Training the classifier jointly with the generator and discriminator requires more time, and is inefficient since in the early iterations the generator network produces poorly generated images that are not similar to the real ones. While it is possible to use L,2 or other distance- based metrics to regularize the generator by comparing embeddings between real and generated images computed using the encoder, it requires having predefined pairs, and since our goal is to develop generative model, we use the pre-trained classifier to regularize the generator. We use a one-layer linear classifier for classification. As an input, we features extracted using the pre- encoder. When training the AFHQ dataset cross-ente cross-entropy loss, since each image has one attribute per image, when training on CelebA dataset, use the binary cross-entropy loss, since each image has multiple attributes per image.
the
4

---

Under review as a conference paper at ICLR 2022
Toin
75
18-
23
0.75
-30
6.99
-75
-120
Figure 2: 2D t-SNE of the computed features from the AFHQ dataset. Color represents the class in the dataset: yellow cat; green dog; dark violet wild animal.
3.3 CLASSIFICATION REGULARIZATION
Classification regularization is used to force the generator generate images conditionally. For every image, that is generated by generator with given input attributes, the embeddings are computed using the pre-trained encoder, and then the embeddings are classified using the pre-trained attributes' classifier. Generator weights are updated so that attributes predicted by the attribute classifier are the same as the input attributes. For the AFHQ dataset with one attribute per image, we use the cross- entropy loss and for the CelebA dataset with multiple attributes per image, we use the binary cross- entropy loss. Unlike other conditional generation methods, which apply classification regularization at each iteration, we apply the classification at each n-th iteration, since it allows to balance between the generation of diverse samples and the generation frequency is selected for each experiment with the dataset and attributes. Unlike other methods, where generator weights are updated on all losses at once, we update the generator on each loss separately, the adversarial loss would not saturate the classification loss.
the samples with specified attributes. The
3.4 CLUSTERS
propose two approaches for selecting the dataset attributes using: (i) "external" attributes either provided with dataset annotated manually, (ii) "inner" attributes. assigned by using K-means (Lloyd (982)) clustering on features extracted using the pre-trained encoder. The 2D t-SNE of the AFHQ dataset features computed using the pre-trained encoder with the "external" attributes from the dataset shown in Figure 2 There are more than distinct clusters and some of the images from the class of wild animals semantically closer the images from other classes. That means that the pre-trained encoder produces semantic similarity. In Figure 2D t-SNE features from the AFHQ dataset with the different numbers of the "inner" attributes are shown.
EXPERIMENTS
In this section, we describe the experiments and the results obtained for the ablation studies. For the evaluation, we use 3 performance metrics: Fréchet inception distance (FID) (Heusel et al. (2017)) ). which is used compare the distribution of generated images with the distribution real images, inception score (IS) (Salimans et al. (206))) which is an objective metric for evaluating the quality of generated images and Chamfer distance (Ravi et al. (2020). which calculates the distance between features of the images. To compute the Chamfer distance, we compute features of the real and generated image by the pre-trained encoder, then compute the 3D 3-t-SNEs of these features, which are used to compute the Chamfer distance. To determine whether the conditional generated images obey the needed attributes, we use attribute control accuracy. The attribute control accuracy is computed the percentage of the images for which the output of the attribute classifier is the same an input attribute. The attribute control accuracy measures how good the generator is at conditionally generating samples.
5

---

Under review as a conference paper at ICLR 2022
Train Clustors:
Toin Clusters
75
50-
25
0.75
23
-50
0.0
-90
-75
-75
05
-100
clusters
(2) 4 clusters
50
25
4-
-25
-100
-10
(3) clusters
(4) 15 clusters
Figure 3: 2D TSNE AFHQ dataset with different numbers clusters.
4.1 EIGENGAN
We compare the proposed InfoSCC-GAN with the "vanilla" EigenGAN (He et al. (2021)) on the AFHQ dataset. Our model is based the same generator while using different inputs and con- ditional regularization. In the current setup. EigenGAN has 6 layers each with 6 dimensions that are used for interpretable and controllable features exploration. The "vanilla" EigenGAN achieves FID score of 29.02 and IS of 8.52 after 200000 training iterations on AFHQ dataset using global discriminator and Hinge loss (Lim & Ye (217))
The EigenGAN does not allow for terpretable feature exploration for the wild animal images. It can be explained by the imbalance since the "wild" animals class includes multiple distinct subclasses such tiger, lion, fox, and wolf, which are not semantically close. The interpretable dimensions of the "vanilla" EigenGAN trained on AFHQ dataset are shown in Appendix B For visualization purposes, we only show the layers and dimensions with visually observable interpretations.
4.2 CONDITIONAL GENERATION
We achieve the best FID score of 11.59 and IS of 11.06 using the InfoSCC-GAN approach after 200000 training iterations using Patch discriminator (Isola al. (2017) and LSGAN (Mao et al. (017)) loss T22 13). In the current setup, we have 6 layers with 6 explorable dimensions. We use cluster classes selected as 3 main clusters using K-means clustering from embeddings computed by the pre-trained encoder. For visualization purposes, the only shown layers are the ones with visually observable interpretations. The interpretable dimensions the proposed InfoSCC-GAN are shown in Appendix
4.3 ABLATION STUDIES
In this section, we describe the ablation studies we have performed on the type of discriminator, the discriminator loss, and the number of clusters in the dataset.

---

Under review as a conference paper at ICLR 2022
Table Discriminator ablation studies
Discriminator
Loss
FID
IS f
Chamfer distance
Global
Hinge
13.08
10.71
4030
Global
Non saturating
25.62
10.33
28595
Global
LSGAN
29.02
9.89
45583
Patch
Hinge
13.95
10.51
7327
Patch
Non saturating
14.83
10.21
5114
Patch
LSGAN
11.59
11.06
3645
4.3.1 DISCRIMINATOR ABLATION STUDIES
In this section, we describe the discriminator and loss ablation studies. For experiments, we use the AFHQ dataset with "inner" clusters. We compare two discriminators: global discrimina- tor and patch discriminator. The global discriminator outputs one value that the probability of the image being real. The architecture of the global discriminator is inspired by the Eigen- GAN paper. The patch discriminator outputs a tensor of values that represent the probability of the image patch being real, the architecture of the patch discriminator inspired by the pix2pix GAN (Isola et al. (2017)) We compare these discriminators in combination with discrimina-
13 The results of the studies are presented in the Table. [ The visual results are presented in the supplement https://amt https///nnymmuus..m 4open. www.sceeeeeeeeeeeeee InfoSCC- GAN-D113
tor losses: Hinge loss 88 9). non-saturating loss(T0 [1] and LSGAN loss
111-11113//DDEEEE README .ma| The 2D
1-SNEs computed from the embeddings are shown in Figure 4 For all of the discriminators and losses, used the study, the attribute control accuracy is the range 99-100%.
In the formulas below å = Pe, (E y) is generated image, Dxx is the discriminator.
The Hinge discriminator loss is defined as following:
DD.. (xx( [max(() (0, (xx(())) + (es [max(0, Dxx (
(8)
The Hinge generator loss is defined as following:
free
Eqg, [m 1xxxx)))))
(x))).
The non-saturating discriminator loss is defined as following (Lucic et al. (2018):
D (x)) [log((xxxx))) (xxx Egs, [log(1 Dxx (i))
(10)
The non-saturating generator loss is defined as following:
Epez -Egs (e) [Dxx (x))
(11)
The LSGAN discriminator loss is defined as following:
Lo = (xpx(x) [xxxx 1) + Ens. (6) (Dxx(#)²]
(12)
The LSGAN generator loss is defined as following:
Eyez = - E (e) [(Dxs - 1)
(13)
4.3.2 NUMBER OF CLUSTERS ABLATION STUDY
First, we compare the performance of the InfoSCC-GAN for the AFHQ dataset with the "external" attributes and with the "inner" attributes the same number of clusters as classes in "external" attributes. The results of the comparison study are shown in Table 2 The attribute control accuracy metrics are not included the table, since they are 100% for both types attributes. As is shown in Table 2 achieve better performance with the "inner" attributes for all of the metrics: FID, IS, and Chamfer distance. can be explained by the fact that when selecting the "inner" attributes the images with the same attributes semantically closer.

---

Under review as a conference paper at ICLR 2022
SNE the modele
T-SNE the model
100
50
25
25
-92
-75
-103
25
15
-5
30
(1) Global discriminator, Hinge loss
(2) Patch discriminator, Hinge loss
Chamfer distance: 4030
Chamfer distance: 7327
T-SNF
25
-9
-25
-30
-15
(3) Global discriminator, Non saturating loss
(4) Patch discriminator, Non saturating loss Chamfer distance: 5114
Chamfer distance: 28595
75
30
-55
-50
-15
(5) Global discriminator, Non LSSANN loss Chamfer distance: 45583
(6) Patch discriminator, ISSAN loss Chamfer distance: 3645
Figure 4: 2D TSNE of AFHQ dataset and conditionally generated samples using different methods. On each plot cat images from the dataset; 1 dog images from the dataset; 2 wild images from the dataset; 3 generated cat images; 4 generated dog images; 5 generated wild images.
Table 2: Comparison of "external" and "inner" attributes
Attribute type
FID
IS
Chamfer distance |
Inner
13,08
10.71
4030
External
14.30
10.11
4423
In this section, compare generation results InfoSCC-GAN the AFHQ dataset, when work- ing with different numbers of clusters. By default, we use 3 clusters. We compare image generation when selecting 4, 6, and 15 clusters. For the experiments, train the generator with global dis-
8

---

Under review as a conference paper at ICLR 2022
Table 3: Number of clusters ablation study
Number of clusters
FID
IS f
Chamfer distance
3
13.08
10.71
4030
97.37
6.77
44113
148.7
5.45
94858
15
101.4
7.15
54016
Table 4: Conditional generation results on CelebA dataset with selected attributes
Attribute Control Accuracy
FID
IS
Bald
Eyeglasses
Mustache
Wearing Hat Wearing Necktie 98.62%
27.84
9,91
93.27%
99.88%
95.68%
94.62%
criminator, with Hinge loss for 200000 iterations. The comparison results of the numbers of clusters are presented in T we also compute an attribute control accuracy for each experiment, and for all of them the metrics are 99-100%. The best generation results are achieved when using 3 "inner" classes, which can be explained by the balanced number of samples in each class approximately 5000 semantically similar images in each class, since classes were selected by clustering the embeddings.
4.4 CELERA EXPERIMENTS
We run the experiments with CelebA dataset with the different numbers of attributes: 5, 10, and 15 attributes. We have selected the attributes. which can easily and unambiguously distinguished. so we can perform a visual check of how the proposed conditional generation method performs. For the evaluation, we use the FID score, IS, and attribute control accuracy metrics. We have used Chamfer distance for the embeddings computed using the pre-trained encoder, since the embeddings are very dense and do not have visible inner cluster structure, but they are still explorable as is shown in Figure 5 The results on the conditional generation using InfoSCC-GAN with 5 selected When training the model, we have used discrete binary attributes, but we have discovered that the model is able generate from continuous input attributes from [0. 1] range an interpretable way, meaning that model learns to apply each attribute to a different degree, depending the input attribute, if value is closer 1, the effect of attribute is bigger. This feature explored in both CelebA demos. We show that proposed InfoSCC-GAN is able to stochastically conditionaly generate samples with multiple attributes. Results with 10 and 15 attributes are shown in the Appendix D
attributes are shown in Table
5 CONCLUSIONS
In this paper, we propose a novel stochastic contrastive conditional GAN InfoSCC-GAN, which produces stochastic conditional image generation with an explorable latent space. We provide the information information-thhorettiaal theoretical formulation of the proposed system. Unlike other contrastive image genera- tion approaches, our method is truly a stochastic generator, that controlled by the class attributes and by the set of stochastic parameters. We apply novel training methodology based on using pre- trained unsupervised contrastive encoder and a pre-trained classifier with every nith iteration using information-thhorrtiiall interpretation of the proposed
classification regularization. We propose system. We propose a novel attribute selection approach based on clustering embeddings computed using encoder. The proposed model outperforms "vanilla" EigenGAN AFHQ dataset, while it also provides conditional image generation. We have performed ablations studies to determine the best setup for conditional image generation. Finally, we have performed experiments AFHQ and CelebA datasets.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
Yunjey Choi, Youngjung Uh. Jacjun You, and Jung-War Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8185-8194, 2020.
Grill, Florian Strub, Florent Altch'e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. ArXiv, abs/2006.00733 2020.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
Zhenliang He, Meina Kan, and S. Shan. Eigengan: Layer-wise eigen-learning for gans. ArXiv,
abs/2104. 1221104.1476.. 12476, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In NIPS, 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 5967-5976, 2017.
Mingu Kang and Jaesik Park. Contrastive generative adversarial networks. ArXiv, abs/2006.12681 2020.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution us- ing generative adversarial network. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 105-114, 2017.
Jae Hyun Lim and J. C. Ye. Geometric gan. ArXiv, 0bbs/1705.02899 2017.
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
S. Lloyd. Least squares quantization in pem. IEEE Transactions on Information Theory, 28(2): 129-137, 1982. doi: 19.1109/TT..182.1105666
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study. In NeurIPS, 2018.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2813-2888 1-21, 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. ArXiv, abs/1441.11844, 2014.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. ArXiv, abs/ 802.05637, 2018.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In ICML, 2017.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv:2007.0850/ 2020.
Tim Salimans, I. Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Im- proved techniques for training gans. In NIPS, 2016.
10

---

Under review as a conference paper at ICLR 2022
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objec- tive. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.). Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. 661 waw.aaaaeaeeatt......a.aaaaaaaaa
URL. httppss//prooceedingg.... neuripss.....ssss..ee.......... http:riss.ccppppp//eaaae cc/paaper/2016/111e
pdf]
Aäron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. ArXiv, abs/ (bb//8807,3378, 2018.
Jure Zbontar, Li Jing, Ishan Misra, Yann André LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. ICML, 2021.

---

Under review as a conference paper at ICLR 2022
A 2D TSNE OF CELEBA FEATURES COMPUTED USING ENCODER
Bald
(2) Black hair
Blond hair
(4) Blurry
(5) Brown hair
(6) Chubby
(7) Double chin
(8) Eyeglasses
(9) Goatee
(10) Gray hair
(11) Male
(12) Mustache
(13) Narrow eyes
(14) Palc skin
(15) Receding hairline
(16) Rosy cheeks
(17) Sideburns
(18) Wearing hat
Figure 5: 2D TSNE of CelebA features computed using encoder.
12

---

Under review as a conference paper at ICLR 2022
B "VANILLA" EIGENGAN GENERATION RESULTS ON AFHQ DATASET
L2 DO Background color
L2 Scale
L3
Color, from black to white
L2 D1 Color, Orientation
Cat
Dog
D4 Fur
Orientation
L3 color
L3 D5 Shadow
L2 D3 Scale
L3 D2 Fur color, Mouth open
L4 DO Color from blue yellow
L4 D1 Fur shade
Figure 6: Interpretable dimensions in "vanilla" EigenGAN
C INFOSCC-GAN CONDITIONAL GENERATION RESULTS ON AFHQ DATASET
LO DO Animal type
LO D4 Pose, Animal type
L1 D3 Shadow
Lo D1 Animal type
DO Background, Animal type
Light
LO D2 Orientation, Animal type
L1 Fur color, Animal type
DO Background
Scale, Animal type
D2 Background
Ls D1 Background
Figure 7: Interpretable dimensions in InfoSCC-GAN
13

---

Under review as a conference paper at ICLR 2022
D
INFOSCC-GAN CONDITIONAL GENERATION RESULTS ON CELEBA DATASET
Table 5: Conditional generation results on CelebA dataset with selected attributes
Attribute Control Aceuracy?
FID
IS1
Bald
Black Hair
Blond Hair
Brown Hair
Double Chin
32.39
9.04
89.74%
89.61%
86.86%
85,55%
84.82%
Eyeglasses 99.6%
Gray Hair 81.71%
Mustache 92.27%
Wearing Hat
Wearing Necktie
92.83%
89.26%
Table 6: Conditional generation results on CelebA dataset with selected attributes
Attribute Control Accuracy
FID
1ST
Bald
Blurry
Chubby
Double Chin
Eyeglasses
34.97
8.87
83.6%
46%
80.1%
95.74%
98.11 11.1%% %
Goatee
Gray Hair
Mustache
Narrow Eyes
Pale Skin
89.09%
90.78%
87.64%
74.22%
86.91 %
Receding Hairline
Rosy Chicks 78.88%
Sideburns 74.9%
Wearing Hat
Wearing Necktie 94.87% %
86.46%
97.64%
14


Reviews:
[{"review_id": "tMgw00YWyjw", "reviewer": "Reviewer_bPdp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The authors interpret their proposed method from an information-theoretic perspective, which makes a better theoretical basis for the paper.\n2. The authors cluster the latent space embeddings extracted by the encoder as new attribute labels, which is novel.\n\nWeaknesses\n1. For classification regularization, the authors didn't indicate the selection method and the precise value of n when applying classification. As you mentioned in section 3.3, “the frequency is selected for each experiment with the dataset and attributes”, does that mean you need to run many times for every dataset and attribute? Since it is one of your contributions, I think more details are needed here.\n2. You only compare your method with EigenGAN on the AFHQ dataset, more conditional image generation methods should be compared.\n3. I don’t know the purpose of the ablation study on the discriminator. You only compare the existing structure of discriminator and loss functions to choose a better combination, which does not reflect the importance of the components of your framework. I think you should compare the encoder of yours with others to show your superior of the framework.\n4. In table 3, the performance is much worse when the cluster is bigger than 3. I think if the encoder produces semantic similarity, then it can cluster better for wild animals when the num is bigger than 3, thus improving GAN training. From Table 3 and Figure 5, I don’t think the encoder produces a good latent space.\n", "summary_of_the_paper": "This paper proposes a stochastic contrastive conditional GAN, which consists of an encoder, an attributes’ classifier and a stochastic EigenGAN generator. The encoder learns to extract the features into a latent space based on contrastive learning. It can provide internal attributes of the dataset for training. The classifier is trained to guide the generator to generate images with the corresponding attributes. The EigenGAN generator guarantees to generate stochastic images. Experiments on AFHQ and CelebA datasets show the effectiveness of the proposed method.", "summary_of_the_review": "Details of the proposed method and experiments are important for readers to follow the main concepts. It would have been better to provide more details in the paper. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "s8kZu66Sp7Q", "reviewer": "Reviewer_atoU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed method includes a contrastive encoder that can mount existing contrastive losses.\n2. For the CelebA dataset, the authors show that InfoSCC-GAN can generate images conditioned on multiple attributes simultaneously.\n\nWeaknesses:\n1. The technical novelty of the paper is limited. InfoSCC-GAN introduces a classifier that minimizes cross-entropy, which is the same as AC-GAN [1] or ClusterGAN [2]. I think the difference is the pre-training of the contrastive encoder, which is just an application of self-supervised learning methods.\n2. A method for selecting inner attributes using K-means clustering on features already exists [3]. The novelty here is that InfoSCC-GAN pre-trains the encoder via contrastive loss.\n3. Major improvements are expected in the experiments section. This work is not the first paper for image generation conditioned on either external and inner attributes. Therefore, more comparative studies are needed to support the advantages of the proposed method. In addition, the paper does not provide generated CelebA images. If space is not enough, t-SNE can be resized or moved to the appendix.\n4. The authors mentioned in the introduction that inner attributes are useful for unbalanced datasets. When learning unbalanced attributes on the AFHQ dataset (e.g., the number of clusters = 6), we can expect that the dataset is clustered with six distinct inner attributes (e.g., cat, dog, tiger, lion, fox, wolf). However, FID increases significantly, which means that the image quality is severely compromised. I think there is a contradiction between the authors' argument and the experimental result.\n\nQuestions:\n1. What is the resolution of the images used in the experiment?\n2. In stage 3, the input space of the generator and the output space of the encoder share the same notation ($\\epsilon$). Is there any constraint to make the two spaces have the same meaning?\n3. When selecting the inner attributes on unbalanced datasets, is $p_y(y)$ determined from a result of clustering?\n4. I wonder if the correlation between attributes (e.g., Male & Beard & Bald …) is considered when conditioning multiple attributes in the CelebA dataset. Could you provide any relevant experimental results?\n\n[1] Conditional Image Synthesis with Auxiliary Classifier GANs, ICML 2017.\\\n[2] ClusterGAN: Latent Space Clustering in Generative Adversarial Networks, AAAI 2019.\\\n[3] Diverse Image Generation via Self-Conditioned GANs, CVPR 2020.\n", "summary_of_the_paper": "This paper proposes a GAN for conditional generation. The authors combine an unsupervised contrastive encoder, stochastic EigenGAN generator, and a classifier. InfoSCC-GAN can perform image generation conditioned on external attributes by maximizing mutual information between input data and class attributes. In addition, clustering on the output space of the pre-trained encoder allows us to generate images conditioned on inner attributes.", "summary_of_the_review": "In my opinion, this submission is not yet ready for publication. The novelty of the proposed method is limited, and more thorough experiments are needed (Please refer to main review).\n\n------------------------------------------------------------------------------------\n\nUpdate:\n\nI will keep my score unchanged because the authors have not provided a rebuttal so far.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "p2u4OQo75e", "reviewer": "Reviewer_wdWF", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have multiple major concerns on this paper. First, the idea is not novel, and the mutual information-based objectives do not provide much insights about the idea. Second, this work misses necessary ablations to showcase the advantage of the proposed design. The reported ablation on the discriminator loss and architecture is rather secondary. It is more interesting to see how the classifier design affects the results, e.g., the benefit of having used the SimCLR as the backbone, and only adding the regularization term every $n$ iterations. Third, this work misses comparison with the literature in conditional image synthesis. It is definitely not enough to just compare with EigenGAN. Even the comparison with EigenGAN is very limited. The reported numbers are not strong. For instance, in Tab. 3, the FID drops significantly by changing from 3 to 4 clusters. This indicates that the whole training pipeline is quite fragile.", "summary_of_the_paper": "The authors aimed to improve EigenGAN for conditional image generation. The conditional information can be obtained via explicit supervision or clustering. The whole idea is not real novel. As already appeared in InfoGAN (2016), a classifier was introduced to facilitate the conditional image generation. Here, the authors suggested to use a pretrained classifier which uses a SimCLR based encoder as the backbone. As the regularization term, the classification loss is active every $n$th iteration. However, in the paper, there are no obvious evidence for the superiority of such design choice.\n", "summary_of_the_review": "Based on my main review comments, I suggest \"reject\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for conditional generations for GANs.
The reviewers note the lack of novelty, or the lack of a theoretical or empirical motivation for the novel bits. They point out flaws in the correctness of the paper, and limited experimental evaluation. 
The reviewers agree to reject the paper. Unfortunately the authors did not answer the reviewers. I therefore recommend to reject the paper for this conference, and I strongly suggest that the authors address the reviewers concerns if they are to submit this paper again in a future venue.
Decision: Reject