Contrastive Fine-grained Class Clustering via Generative Adversarial Networks
Authors: Yunji Kim
Jung-Woo Ha
Venue: ICLR 2022 Spotlight (2022)
Keywords: Unsupervised Fine-grained Class Clustering
Disentangled Representation Learning
Generative Adversarial Networks

Abstract: Unsupervised fine-grained class clustering is a practical yet challenging task due to the difficulty of feature representations learning of subtle object details. We introduce C3-GAN, a method that leverages the categorical inference power of InfoGAN with contrastive learning. We aim to learn feature representations that encourage a dataset to form distinct cluster boundaries in the embedding space, while also maximizing the mutual information between the latent code and its image observation. Our approach is to train a discriminator, which is also used for inferring clusters, to optimize the contrastive loss, where image-latent pairs that maximize the mutual information are considered as positive pairs and the rest as negative pairs. Specifically, we map the input of a generator, which was sampled from the categorical distribution, to the embedding space of the discriminator and let them act as a cluster centroid. In this way, C3-GAN succeeded in learning a clustering-friendly embedding space where each cluster is distinctively separable. Experimental results show that C3-GAN achieved the state-of-the-art clustering performance on four fine-grained image datasets, while also alleviating the mode collapse phenomenon. Code is available at https://github.com/naver-ai/c3-gan.

Published as a conference paper at ICLR 2022
CONTRASTIVE FINE-GRAINED CLASS CLUSTERING VIA GENERATIVE ADVERSARIAL NETWORKS
Yunji Kim NAVER Lab
Jung-Won Woo Ha NAVER AI Lab & NAVER CLOVA
yunji menjj..ki@@aavercco.
jungwoo- hac waweecorrp........ com
ABSTRACT
Unsupervised fine-grained class clustering is a practical yet challenging task due to the difficulty of feature representations learning of subtle object details. We introduce C3-GAN, a method that leverages the categorical inference power of InfoGAN with contrastive learning. We aim to learn feature representations that en- courage dataset to form distinct cluster boundaries in the embedding space, while also maximizing the mutual information between the latent code and its image ob- servation. Our approach is to train a discriminator, which is also used for inferring clusters, to optimize the contrastive loss, where image- the mutual information Specifically, we map the input of a generator, which was sampled from the categor- ical distribution, to the embedding space of the discriminator and let them act as a cluster centroid. In this way, C3-GAN succeeded in learning a cuustering-frenndie embedding space where each cluster distinctively separable. Experimental results show that C3-GAN achieved the state-of saute-of-the tit-hhh-rttt art clustering performance on four fine-grained image datasets, while also alleviating the mode collapse phenomenon. Code is available at Pnttps://ge thub. com/ http//vvverraaaaa aitteerraai//-aaaaa
pairs that maximize considered positive pairs and the rest negative pairs.
INTRODUCTION
Unsupervised fine-grained class clustering is a task that classifies images of very similar ob- jects (Benny & Wolf 2020). While a line of works based on multiview-based self-supervised learning (SSL) methods (He et al. 2020: Chen et al. 2020; Grill et al. 2020) shows promising results on a conventional coarse-grained class clustering task (Van Gansbeke al., 2020). it is difficult for a fine-grained class clustering task to benefit from these methods for two reasons. First, though it is more challenging and ambiguous for finding distinctions between fine- fine-grained classes, datasets for this type of task are difficult to be large-scale. We visualized this idea in Figure [ which shows that finding distinctions between fine-grained classes is more difficult than doing so for coarse-grained object classes. Second, the augmentation processes in these methods consider subtle changes in color or shape, that actually play an important role differentiating between classes, as noisy factors. Thus, it is required to find an another approach for mnaigra class clustering task.
Generative adversarial networks (GAN) (Goodfellow et al., 2014) can be a solution as it needs to learn fine details to generate realistic images. A possible starting point could be InfoGAN (Chen et al. 2016), that succeeded in unsupervised categorical inference on MNIST dataset (LeCun et al. 2010) by maximizing the mutual information between the latent code and its observation. The only prior knowledge employed was the number of classes and the fact that the data is uniformly distributed over the classes. FineGAN (Singh al. (019)) extends InfoGAN by integrating the scene decomposition method into the framework, and learns three latent codes for the hierarchical scene generation. Each code, that is sampled from the independent uniform categorical distribution, is sequentially injected to multiple generators for background, a super-class object, and a sub-class object image syntheses. FineGAN demonstrated that two latent codes for object image generations could be also utilized for clustering real images into their fine-grained classes, outperforming conventional coarse-grained class clustering methods. This result implies that extracting only foreground features is very helpful for the given task. However, FineGAN requires object bounding box annotations and additional training of classifiers, which greatly hinder its applicability. Also, the method lacks the ability of learning the distribution of real images, due the mode collapse phenomenon (Higgins al. 2017).

---

Published as a conference paper at ICLR 2022
(a) Fine-grained Class Clustering
(b) Coarse-grained Class Clustering
Coarse-grained. We compare two sets of 4 classes, each selected from (a) CUB (Wah et al., (01T)) and (b) STL (Coates et al. 201I) datasets. The diagrams visualize image features extracted from R that was trained on each dataset via the method of SimCLR (Chen ett al. 2020). The color of each dot corresponds to its ground-trutt class. Both in the image-level and the feature-level, the clusters in (a) are way more indistinguishable than those in (b).
Figure 1: Fine-grained
In this paper, we propose Constrastive fine-grained Class Clustering GAN (C3-GAN) that deals with the aforementioned problems. We first remove the reliance on human-annotated labels by adopting the method for unsupervised scene decomposition learning that is inspired by PerturbGAN (Bielski & Favaro 2019b. We then adopt contrastive learning method (van den Oord et al. 2018) for training a discriminator by defining pairs of image-laten features that maximize the mutual information of the two positive pairs and the rest negative pairs. This is based on the intuition that optimal results would be obtained if cluster centroids are distributed in a way that each cluster can be linearly separable (Wang & Isola 2020). In specific, we map the input of a generator that was sampled from the categorical distribution onto the embedding space of the discriminator, and let it act as a cluster centroid that pulls features of images that were generated with its specific value. Since the relations with other latent values are set as negative pairs, each cluster would be placed farther away from each other by the nature of the contrastive loss. We have conducted experiments on four fine- fine-grained image datasets including CUB, Stanford Cars, Stanford Dogs and Oxford Flower, and demonstrated the effectiveness of C3-GAN by showing that achieves the state-of-the-an clustering performance on all datasets. Moreover, the induced embedding space of the discriminator turned out to be also helpful for alleviating the mode collapse issue. We conjecture that this is because the generator additionally required to be able synthesize certain number of object classes with distinct characteristics.
Our main contributions are summarized as follows:
We propose a novel form of the cettmmaaiionthhooette regularization to learn a clustering- friendly embedding space that leads dataset to form distinct cluster boundaries without falling into a degenerated solution. With this method, our C3-GAN achieved the state-of- the-art fine-grained class clustering performance on four fine-grained image datasets.
By adopting scene decomposition learning method that does not require any human- annotated labels, our method can be applied to more diverse datasets.
Our method of training a discriminator is not only suitable for class clustering task, but also good alleviating the mode collapse issue of GANs.
RELATED WORK
Unsupervised Clustering. Unsupervised clustering methods can be mainly categorized into the information theory-based and
approaches. The first approach aims to maximize the mutual information between original images and their augmentations to train model in end-to-end fashion (Ji al. 2019: Zhong et al., 2020). To enhance the performance, IIC (Ji et al., 2019) additionally trains the auxiliary branch using an unlabeled large dataset, and DRC (Zhong et al. 2020) optimizes the contrastive loss on logit features for reducing the intra-class feature variation. EM-based methods decouple the cluster assignment process (E-step) and the feature representations learning process (M-step) to learn more robust representations, Specifically, the
2

---

Published as a conference paper at ICLR 2022
feature representations are learnt by not only fitting the proto-clustering results estimated in the expectation step, but also optimizing particular pretext tasks. The clusters are either inferred by k-means clustering (Xie et al. [016 Caron et al. 2018 Liu et al. 2020 Li et al. (222)) or training of an additional encoder (Dizaji et al. 2017). However, many of these methods suffer from an uneven
allocation issue of k-means c clustering. which could result in degenerate solution. For this reason, SCAN (Van Gansbeke et al. (020) proposes the novel objective function that does not require the k-means clustering process. Based on the observation that K nearest neighbors of each feature point belong to the same ground-truth cluster with a high probability, it trains classifier that assigns identical cluster id to all nearest neighbors, which significantly improved the clustering performance. Unlike these prior methods where the cluster centroids are not learnt in end-to-end manner or not the subject of interest, C3-GAN investigate and try to improve their distribution for achieving better clustering performance.
Unsupervised Scene Decomposition. Heavy cost of annotating segmentation masks have drawn active research efforts on developing unsupervised segmentation methods. Some works address the task in the information-theorel the works are based on GANs. ReDO (Chen et al. (2019)) learns to infer object masks by redrawing an input scene, and PerturbGAN (Bielski & Favaro (2019), that has a separate background and foreground generator, triggers scene decomposition by randomly perturbing foreground images. Meanwhile, another line of works utilizes pre-trained high quality generative models such as Style-
perspective (Ji et al. 2019; Savarese et al. 202T), but a majority of
GAN (Karras et al., 2019: 2020) and BigGAN (Brock et al.) 2019). Labels4Free (Abdal al. 2021) trains a segmentation network on top of the pre-trained StyleGAN, utilizing the fact that imputs of each layer of StyleGAN have different degrees of contribution to foreground synthesis. Voynov & Babenko (020) and Melas- Kyriazi et al. (202T) explore the latent space of pre-trained GANS to find the perturbing directions that can be used for inducing foreground masks. While all these methods additional mask predictor with synthesized data, or projecting real images into the latent space of the pre-trained GAN, C3-GAN is rather focusing on the semantic representations learning of foreground objects.
aim to infer foreground masks by training
Fine-grained Feature Learning. For fine-graing gree-graied feature representations learning, some works (Singh et al. 2019 Li et al. 2020 Benny & Wolf 2020) have extended InfoGAN by integrating the scene decomposition learning method into the framework with multiple pairs of adversarial networks. FineGAN (Singh al.) 2019) learns multiple latent codes to them for sequential generation of a background, a super-class object, and a sub-class object image, respectively. MixNMatch (Li al. 2020) and OneGAN (Benny & Wolf, 2020) extend FineGAN with multiple encoders to directly infer these latent codes from real images and use them for manipulating images. Since this autoeneoder-based structure could evoke a degenerate solution where only one generator is responsible for synthesizing the entire image, MixNMatch conducts adversarial learning on the joint distribution of the latent code and image (Donahue et al. 2017), while OneGAN trains model in two stages by training generators first and encoders next. Even though these works have succeeded in generating images a hierarchical manner and learnt latent codes that can be used for clustering real images corresponding to their fine-grained classes, they cannot be applied to datasets that have no object bounding box annotations. Moreover, they require a training of additional classifiers using set of generated images annotated with their latent values. The proposed model in this paper, C3-GAN, can learn the latent code of foreground region in a completely unsupervised way, and simply utilize the discriminator for inferring the clusters of given dataset.
3 METHOD
Given dataset X= = (z, 1== consisting of single object images, we aim distribute data into Y semantically -raiied classes. Our GAN-based model infers the clusters of data in the semantic feature space H E d of the discriminator D. The feature space H. is learnt by maximizing the mutual information between the latent code, which is the input of a generator, and its image observation é. For more robust feature representations learning, we decompose a scene into background and foreground region and associate the latent code mainly with the foreground region. We especially reformulate the information -heoortiin regularization to optimize the contrastive loss defined for latent- -mage feature pairs to induce each cluster to be linearly separated in the feature space.
3

---

Published as a conference paper at ICLR 2022
CLUSTER CENTROIO FEXTURE (3)
CLUSTER
c
000
V-13
D pese
&R
gTa!
D base
CLUSTER
w(o.))
Ranken
5M
FARST
CLUSTER
g
CLUSTER
Figure 2: Overview of C3-GAN, It synthesizes a background image bby and a foreground image f/9 from the background generator Gh9 and the foreground generator G respectively. To trigger this decomposition, we perturb foreground image with the random affine transformation matrix Ta right before the composition of two image components. The association between the latent code e and its image observation E is learnt by optimizing the information-thhoreee based on the contrastive loss defined for their feature representations in the embedding space of the discriminator. The inference of fine-grained class clustering is made based on the distances between images' semantic features )h:(0)) N-1 that are depicted with dotted lines, and the set of cluster centroids {4( ))
regularization that is
i= Y-1 which are the fixed number of embedded latent codes c.
3.1 PRELIMINARIES
Our proposed method built on FineGAN (Singh et al. 2019), which is in turn based on InfoGAN (Chen et al. 2016). InfoGAN learns to associate the latent code with its observation è by inducing the model maximize the mutual information of the two, 1(c, f). The latent code e can take various form depending on the prior knowledge of the factor that we want to infer, and is set to follow the uniform categorical distribution when our purpose is to make the categorical inference on given dataset. FineGAN learns three such latent codes for hierarchical image generation, and each code respectively used for background, super-class object, and sub-class object image synthesis. To facilitate this decomposition, FineGAN employs multiple pairs of generator and discriminator and trains an auxiliary background classifier using object bounding box annotations. It further demonstrates that the latent codes for object image synthesis can be also utilized for clustering an image dataset according to their fine-grained classes.
Our method differs from FineGAN in that, i) it employs the discriminator D infer clusters without requiring additional training of classifiers, and ii) it learns only one latent code c that corresponds the fine-grained class of foreground object. The separate random noise is kept, that is another input of the generator, to model variations occurring in a background region. The noise value 2 E Rdd is sampled from the normal distribution (' I), and the latent code c E RY is an 1-hot vector where the index io that makes C = is sampled from the uniform categorical distribution, (4(0, Y 11) Specifically, the background generator 6o synthesizes a background image 2h E 3 x solely from the random noise and the foreground generator G synthesizes foreground mask m R! x HxW and an object image f R xW using both z and c. To model foreground variations, we convert an 1-hot latent code to variable c' E = whose value is sampled from the gaussian distribution N (Me, G2) where the mean d.. and diagonal covariance matrix a are computed according to the original code (Zhang et al. 2017). The final image & 3 H x is the composition of generated image components summed by the hadamard product, as described in Fig. 2
To achieve a fully unsupervised method, we leverage the scene decomposition method proposed by PerturbGAN (Bielski & Favaro 2019). In specific, scene decomposition is triggered by perturbing foreground components m and with the random affine transformation matrix Te right before the final image composition. The parameters 0 of the random matrix To include rotation angle, scaling factor and translation distance, and they are all randomly sampled from the uniform distributions with predefined value ranges.
In summation, the image generation process is described as below:
à' ~ (NN(ue, oc),
ghg m. f= G(z,c')
(1)
m', i) & -= = o (1 1 mi) + on'.
4

---

Published as a conference paper at ICLR 2022
3.2 CONTRASTIVE FINE-GRAINED CLASS CLUSTERING
We assume that data would be well clustered when i) they form explicitly discernable cluster boundaries in the embedding space H. and ii) each cluster centroid by E² condenses distinct semantic features. This is the exact space that the discriminator of InfoGAN aims to approach when the latent code is sampled from the uniform categorical distribution, (0, Y 1). However, since jointly optimizes the adversarial loss, the model has a possibility of falling into the mode collapse phenomenon where the generator G covers only a few confident modes (classes) to easily deceive the discriminator D. This is the reason why InfoGAN lacks the ability of inferring clusters on real image datasets. To handle this problem, we propose a new form of an auxiliary probability ((clz) that represents the mutual information between the latent code and image observation 2.
Let us now describe the objective functions with mathematical definitions. The discriminator D aims to learn an adversarial feature E IK for the image authenticity discrimination, and a semantic feature IRd for optimizing the information- -heoorettteeeeeeeee regularization. The features are encoded from separate branches t and whs which were split at the end of the base encoder of the discriminator D Dbase The adversarial feature r is learnt with the hinge loss as represented in the equations below:
code [EtxX min(O,, r) + min 1 F) 1 (D basee (x)).
Cade -Ex-G(Z,)) (2)
s.t.
Meanwhile, the semantic feature h can be considered as a logit in a classification model. In a supervised setting where ground-trute -ruth class labels are given, data mapped into embedding space gradually form discernable cluster boundaries as the training proceeds, similar to the one in Fig. (b). However, in an unsupervised setting, it is difficult to attain such space simply by maximizing the mutual information between the latent code and its observation & for the aforementioned reasons. formulate an auxiliary probability Q(c[z) in the form of contrastive loss. We map the latent code onto the embedding space H. with a simple linear layer Ve,, and let it act as cluster centroid (1) that pulls semantic features of images (h) that were generated from its specific value. By setting the relations with other latent codes as negative pairs, cluster centroids are set to be pushing each others and form distinguishable boundaries in the embedding space. Specifically, we expect the semantic feature / to be mapped to the k-th cluster centroid, where k is the index of its corresponding latent code c that makes C = 1, while distances to other centroids ly are far enough to maximize the mutual information of the two. To enhance the robustness of the model and assist the scene decomposition learning, we additionally maximize the mutual information between the latent code e and masked foreground images $19. To up, the proposed information-theorrtie objectives defined as follows:
Our solution
Linfo EE log qk ), exp(sim (h, 4x)T)) h
Linsota = =Bx-G(2,o) - log qh
(3)
s.t. g: Q(ck 1|x) Exo exp(sim (h.1,)/7)
1= (c(lr).
where sim(a, b) is the cosine distance between vectors a and 6, y is an index of cluster, and 7 is the temperature of the softmax function. ly denotes the identity matrix with the size of Y x Y.
Additional Regularizations Following the prior works (ji et al. 2019 Van Gansbeke et al. 2020). we adopt the overclustering strategy to help the model learn more expressive feature set. We also regularize the prediction on a real image q R Y by minimizing its entropy H (q) promote each data to be mapped to only one cluster id with high confidence, along with the minimization of the KL divergence between batch-wisele wisely averaged prediction q and the uniform distribution u, that is for avoiding degenerated solution where only a few clusters are overly allocated. Furthermore, we optimize the contrastive loss (Chen al. 2020) defined for real image features h to assist the semantic feature learning. To summarize, the regularizations real images are as follows:
E -log exp(sim(h, h()/T) ].
Cimt
En= exp(sim(h, ())))
(4)
centropy = E (H) + (qlu) s.t.
q = Ex (g].
5

---

Published as a conference paper at ICLR 2022
We also enforce foreground mask m to be more like hard mask and take up reasonable portion of a scene by employing below regularization function.
mmskk Gx,c() [Rw!
[ [-re log(m)) - (1 m)) log(1 rit] + max (0, 0.1 - [ml(mm] + max(0, Te [mm] 0.9) ].
(5)
In sum, C3-GAN is trained alternately optimizing the following objective functions for D and G:
min f + LG raLinso + Cinfol + +226 A + Cenerropy Aa &aakk (6) D.G
EXPERIMENTS
4.1
EXPERIMENTAL SETTING
Datasets.
We tested our method on 4 datasets that consist of single object images. i) CUB (Wah
ct al.
2011 : 5,994 training and 5,794 test images of 200 bird species. ii) Stanford Cars Krause
ct al.
models. iii) Stanford Dogs KKholla 2011): 12,000 training and 8,580 test images of 120 dog species. iv) Oxford Flower (Nilsback
2013): y 14444 training and 8,041 test images of 196
ct al.
& Zisserman 2008): 2,040 training and 6, 6,149 test images of 102 flower categories. Due to the small
number of traming images, all models for CUB and Oxford Flower datasets were trained with the
entire dataset the prior works did (Singh al. 2019 Li al. 2020 Benny & Wolf 2020).
Implementation Details. The weights of the loss terms (Ao, ÅI, Aę, As, A4) are set as (5, 1, 1, 0.1, 1), and the temperature set as 0.1. We utilized Adam optimizer of which learning rate is 0.0002 and values of momentum coefficients are (0.5, 0.999). The architectural specification of C3-GAN and the values of hyperparameters, such the parameter ranges the random affine transformation and the number of clusters Y can be found in Appendix A. and Appendix A.2 respectively.
Baselines. We first compared C3-GAN with IIC (Ji et al. 2019) in order to emphasize the dif- ference between the coarse graaine and fine- grained class clustering task, since it achieved decent performance with sobel filtered images where color and texture informations are discarded. We also experimented with simple baseline of LimCLR*K-Mest ing instance discrimination task is not enough for fine-grained feature learning. SCAN (Van Gansbeke et al. 2020), the work that shows remarkable coarse-grained class clustering performance, is also compared to investigate whether the method of inducing consistent predictions for all nearest neigh- bors would be helpful for the fine-grained class clustering task as well. Finally, we compared with the methods that learn hierarchically fine tne-grrined feature representations, such as FineGAN (Singh al. 2019). MixNMatch (Li et 2020) and OneGAN (Benny & Wolf, 2020), including their base model InfoGAN (Chen et al. 2016). to claim the efficacy of the proposed formulation that is based on the contrastive loss. We mainly compared with the unsupervised version of them that trained with aaseudo-labeled bounding boxes which assign edges of real images background region.
(Chen al. 2020) claim that optimiz-
4.2 FINE-GRAINED CLASS CLUSTERING RESULTS
Quantitative results. We evaluated clustering performance with two metrics: Accuracy (Acc) and Normalized Mutual Information (NMI). The score of accuracy is calculated following the optimal mapping between cluster indices and real classes inferred by Hungarian algorithm (Kuhn & Yaw 1955) for given contingency table. We presented the results in Table As can be seen, C3-GAN outperforms other methods by remarkable margins in terms of Acc on all datasets. Our method presents better or comparable performance underline that understanding only structural characteristic is not enough for the fine class clustering task. From the results SCAN (2nd rank) and MixNMatch (3rd rank), we can conjecture that both requiring local disentanglement of the embedding space and extracting foreground features via scene decomposition learning can be helpful for improving the fine- mee-grained class clustering performance. However, it is difficult to determine whether the GAN-based methods the SSL- based methods are better because both approaches have similar score ranges for all metrics and display globally entangled feature spaces as shown in Fig. 3 (a) and (b). Since C3-GAN resembles
terms of NMI scroes as well. The results of IIC

---

Published as a conference paper at ICLR 2022
Acc f
NMI
Dog
Dog
FineGAN Singh cl.
2019
12.6
7.8
7.9
0.40 0.35 0.23
MixNMatch
Li al.
2020
13.6
7.9
8.9
0.42 0.36 0.32
OneGAN
Benny Wolf
2020
10.1
6.0
7.3
0.39 0.27 0.21
Fully Unsupervised Setting
IIC at.
2019
5.0
8.7
0.36 0.27 0.18 0.24
SimCLR Chen et al
2020
+k- Means
8.4
6.7
6.8
12.5
0.40 0.33 0.19 0.29
InfoGAN Chen al.
2016
8.6
6,5
6,4
23.2
0.39 0.31 0.21 0,44
FineGAN w/o labels
6.9
6.8
6.0
8.1
0.37 0.33 0.22 0.24
MixhMatch w/o labels
10.2
10.3
39.0
0.41 0.34 0.30 0.57
SCAN Van Gansbeke ct.l.
2020
11.9
8.8
12.3
56.5
0.45 0,38 0.35 0.77
C3- 33-GAN (Ours)
27.6
14.1 17.9 67.8
0.53 0.41 0.36 0.67
Table 1: Quantitative evaluation of clustering performance. t denotes that the values are reported ones in the original papers, and the rest scores are obtained by experimenting with the released codes of baselines set of evaluation datasets. Please note that the original methods of FineGAN, MixNMatch, and OneGAN utilize human-annotated labels.
(a) MixNMatch al. 2020
(b) SCAN
Van Gansheke
2020
(c) C3-GAN (Ours)
Figure 3: Visualization of embedding spaces of MixNMatch, SCAN and C3-GAN that were trained on CUB dataset. this end, we reduced the semantic features of images sampled from 10 classes into 2-dimensional space through t-SNE. This result implies that only C3-GAN has learnt the embedding space where clusters are explicitly separable.
MixNMatch to some extent, its state-of-the-ant scores can be partially explained by the fact that it
learns decomposed foreground features, but the embedding space C3-GAN visualized in Fig. (c) implies that the performance of it has been further improved from our formulation of (c((r) that
enforces clusters be distinetively distributed in the embedding space. We additionally presented the results of qualitative analysis clustered images inferred by the top three methods in Appendix A.3
which better demonstrate the excellence of our method compared to prior works.
blltion Study. We conducted ablation study regarding three factors: Overclustering, ii) Foreground perturbation, and iii) The informatiin--heeretii regularization based on the contrastive loss. We investigated clustering performance of C3-GAN on CUB dataset by removing each factor. The results can be seen in Table The largest performance degradation was caused by changing the formulation of the information-theorett regularization to a trivial softmax function. This result implies that clustering performance has high correlation with the degree of separability of clusters, which was significantly improved by our method. Regarding the result ii), we observed that C3-GAN fails to decompose scene when the foreground perturbation was not implemented. This degenerated result suggests that the method of extracting only foreground features has a non-negligible amount of influence in clustering performance. Lastly, the result of i) implies that the overclustering strategy also helpful for boosting performance since it allows a larger capacity for model to learn more expressive feature set. The additional analysis hyperparameters can be found in Appendix A.2
7

---

Published as a conference paper at ICLR 2022
Acc t
NMI
C3-GAN (Ours)
27.6
0.53
))
Overclustering
22.7
0.50
ii) Foreground perturbation
16.6
0.47
mii
The aeoommtion-tthoretiic regularization based on the contrastive loss
14.5
0.45
Table 2: Result of an ablation study. We observed that the largest performance degradation was caused by changing the formulation of the cntemmation-thhorett regularization trivial softmax function.
The other two factors also have a non negliggie impact on performance.
Real
Fake images
Real
Fake images
Real
Fake images
Background
Background
Background
Foreground
Foreground
Foreground
(a)
Fixed and variable
Variable c and fixed x
(b)
Figure 4: Qualitative analysis of the image generation performance. We present (a) images synthesized with the cluster indices of real images that were predicted by the discriminator, and (b) images synthesized by controlling values of the latent code c and the random noise z.
4.3 IMAGE GENERATION RESULTS
Qualitative Results. For qualitative evaluation, we analyzed the results on CUB dataset. In Fig. (a), displayed synthesized images along with their decomposed scene elements. The images generated with the cluster indices real images that were predicted by the discriminator. Considering the consistency of foreground object features within the same class and the uniqueness of each class feature, we can assume that C3-GAN succeeded in learning the set of clusters in which each cluster represents its unique characteristic explicitly. Further, to investigate the roles of the input variables, and we analyzed the change in the synthesized images when they were generated by fixing one of the two and diversifying the other, as shown in Fig. (b). As it can be seen, if c fixed, all images depict the same bird species, but the pose of bird and background vary depending on the value z. Conversely, when generated under the condition of variable and fixed z, images of various bird species with the same pose and background were observed. The results other datasets show the same trend. Please refer to Appendix A..4
It is also notable that C3-GAN can effectively alleviate the mode collapse issue with the proposed method. Fig. 5 present that only C3-GAN reflects intra-class deviations various factors such as color, shape, and layout for the given class, while other baselines produce images only with the layout variation. We conjecture that this performance was achieved because the learnt embedding space of the discriminator better represents the real data distribution, allowing the generator get quality
8

---

Published as a conference paper at ICLR 2022
Real
FineGAN Singh et 2019)
C3-GAN (Ours)
MixNMatch
(Li al. 2020
Figure 5; Images synthesized with the cluster index that corresponds to the class of real samples. Only C3-GAN generates images reflecting the intra-class color, shape, and layout variations of the real image set.
FID
+
IS 1
Reverse KL |
Bird
Car
Dog
Flower
Bird
Dog
Flower
Bird
Dog
Flower
InfoGAN Chen ot 2016
35.71 70.91 59.07 78.37
8.20 3.25 6.91 13.39
0.56 0.73 0.40 0.42
FineGAN w/o labels
33.87 87.66 49.12 35.72
10.07 4.20 10.67 10.42
0.67 0.58 0,24 0.52
MixNMatch w/o labels
31.59 78.36 48.11 32.03
9.76 4.11 10.70
16.48
0.52 0.57 0.22 0.34
C3-GAN (Ours)
19.37 67.36 45.40 64.19
14.52 3.62 9.33 13.75
0.29 0.57 0.18 0.25
Table 3: Quantitative evaluation of image synthesis performance.
signals for the adversarial learning. Additionally, we found that only C3-GAN succeeded in the scene decomposition learning when annotations were not utilized. In fact, when FineGAN and MixNMatch were trained without object bounding box annotations, they fell into degenerate solutions where the entire image is synthesized from the background generators.
Quantitative Results. We also quantitatively evaluated the image synthesis performance based on scores Frechet Inception Distance (FID) and Inception Score (IS). We also considered the scores of reverse KL which measures the distance between the predicted class distribution of real images and synthesized images. We compared C3-GAN with GAN-based models that were trained without object bounding box annotations. The scores of IS and reverse KI. were measured with Inception networks that were fine-tuned on each dataset using the method of Cui et al. (2018) The results displayed in Table C3-GAN presents the state-of-the-ar terms of all metrics. This result means that, in addition to the class clustering performance, C3-GAN has competitive performance in generating fine iinengraieed object images.
comparable performance in
CONCLUSION
In this study, we proposed new GAN-based method, C3-GAN, for unsupervised fine-grained class clustering which more challenging and less explored than a coarse-grained clustering task. To improve the fine-grained class clustering performance, we formulate the infornation--heerrticc regu- larization based on the contrastive loss. Also, scene decomposition- into the framework to enforce the model to learn features focusing foreground object. Extensive experiments show that our C3-GAN not only outperforms previous fine-grained clustering methods but also synthesizes fine-grained object images with comparable quality, while alleviating mode collapse that previous state-of-thh of-th--rrr GAN methods have been suffering from.
approach incorporated
'The accuracy of Inception Networks fine-tuned CUB/Stanford Cars/Stanford Dogs/Oxford Flower datasets 0.87/0.47/0.8660.94 respectively.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Remarkable advancement of generative models has played a crucial role as Al tools for text (Brown
al. 2020 Kim al. 2021), ) image (Karras et al. 2020 Choi al., 2020b). audio (Sisman et al.
2020), and multimodal content generation (Tian et al. 2021 Yan et al. (02T). However, as its side effect, many malicious applications have been reported as well, thus leading severe societal problems, such as deepfake (Dolhansky et al. 2020), fake news (Lazer al. 2018), and generation biased by training data (Gupta et al. [2021) Our work might be an extension of the harmful effects of generative models. On the contrary, generative models can be solution to alleviate these side effects via data augmentation (Yang et al,, 2019) Choi et al., 2020a). In particular, our method can contribute to alleviating data bias and fine-grained class imbalance. Therefore, many endless efforts are required to make these powerful generative models benefit humans.
ACKNOWLEDGMENTS
This work was experimented on the NAVER Smart Machine Learning (NSML) platform (Sung et al. 2017 Kim al. 2018). We are especially grateful Jun- Jun-Yan Zhu and NAVER Al Lab researchers for their constructive comments.
REFERENCES
Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka, Labels4free: Unsupervised segmenta- tion using stylegan. In ICCV, 2021.
Yaniv Benny and Lior Wolf. Onegan: Simultancous unsupervised learning of conditional image generation, foreground segmentation, and fine-grate ane-grained clustering. ECCV, 2020.
Adam Biclski and Paolo Favaro. Emergence of object segmentation in perturbed generative models. In NeurIPS, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In NeurIPS, 2020.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In ECCV, 2018.
Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by redrawing. In NeurlPS, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML. 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Πly Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NeurlPS, 2016.
Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In ICML, 2020a.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020b.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single layer networks in unsupervised feature learning. In AISTATS, 2011.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Belongie Serge. Large scale fine-grained categorization and domain-specific transfer learning. In CVPR, 2018.
10

---

Published as a conference paper at ICLR 2022
Kamran Ghasedi Dizaji, Amirhossein Herandi, and Heng Huang. Deep clustering via joint convolu- tional autoencoder embedding and relative entropy minimization. In NeurIPS, 2017.
Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cris- tian Canton Ferrer. The deepfake detection challenge (dfde) dataset. In arXiv, 2020.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Jean-Basticn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurlPS, 2020.
Aman Gupta, Deepak Bhatt, and Anubha Pandey. Transitioning from real synthetic data: Quantify- ing the bias model. In Synthetic Data Generation Workshop at ICLR 2021, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick,
Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, 2019.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2020.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for Visual Categorization.
fine-grained image categorization. In First Workshop on 2011.
Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Sconhoon Kim, Dongpil Seo, et al. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. In EMNLP, 2021.
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang. Youngkwan Kim, et al. Nsml: Meet the mlaas platform with a real-world case study, arXiv preprint arXiv: 181.0997, 2018.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition, 2013.
H. W. Kuhn and Bryn Yaw. The hungarian method for the assignment problem. In Naval Research Logistics Quarterly, 1955.
David MJ Lazer, Matthew Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menezer, Miriam J Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, et al. The science of fake news. AAAS, 2018.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. In ATT Labs [Online). Available: http://yaam.eccun.om//e wwtt.//d///e//iistttttmmmmmmmm 2010.
Junnan Li, Pan Zhou, Caiming Xiong. and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR, 2021.

---

Published as a conference paper at ICLR 2022
Yuheng Li, Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Mixnmatch: Multifactor disentanglement and encoding for conditional image generation. In CVPR, 2020.
Steven Liu, Tongzhou Wang, David Bau, Jan Zhu, and Antonio Torralba. Diverse image generation via self-conditioned gans. In CVPR, 2020.
Luke Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Finding an unsupervised image segmenter in each of your deep generative models. In arXiv, 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICCVGIP, 2008.
Pedro Savarese, Sunnie S. Y. Kim, Michael Maire, Greg Shakhnarovich, and David McAllester. Information-theorrtiic segmentation by inpainting error maximization. In CVPR, 2021.
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Finegan: Unsupervised hierarchical disen- tanglement for fine-grained object generation and discovery. In CVPR, 2019.
Berrak Sisman, Junichi Yamagishi, Simon King, and Haizhou Li. An overview of voice conversion and its challenges: From statistical modeling to deep learning. In TASLP, 2020.
Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngkwan Kim, Gayoung Lee, Donghyun Kwak, Hung-Woo Ha, and Sunghun Kim. Nsml: A machine learning platform that enables you to focus on your models. In arXiv, 2017.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski. Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. good image generator what you need for high-resolution video synthesis. In ICL.R, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. In arXiv, 2018.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In ECCV, 2020.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In ICML, 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. In California Institute of Technology, 2011.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In ICMI., 2020.
Junyuan Xie, Ross B. Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, 2016.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. In arXiv, 2021.
Shuo Yang, Kai Shu, Suhang Wang, Renjie Gu, Fan Wu, and Huan Liu. Unsupervised fake news detection on social media: A generative approach. In AAAI, 2019.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
Huasong Zhong, Chong Chen, Zhongming Jin, and Xian-Sheng Hua. Deep robust clustering by contrastive learning. In arXiv, 2020.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 NETWORK ARCHITECTURE
We present detailed description of the architecture of C3-GAN in Table Table and Table dots (.. are for indicating that the outputs of earlier layers are used as the inputs of later layers. The training was done with NVIDIA-VION GPUs, and we optimized model until the convergence of the FID score.
The colored
Module
Layers
Input size
Output size
Linear, BN, GLU, UnSqueeze
512x4x4
Gbg
[Up, Conv(3.1), I), BN, GLU)
512x4x4
64x 128 128
[Conv(3.1), BN, GLU, Conv(3.1), BN
64x 128 128
64x 128 128
Comv(3.1),
128 x
128x112 (8**)
Nijes ae)
Linear, BN, GLU
(c)
Linear, BN, GLU, Reshape
64 (X)
512x4x4
[Up, Conv(3, D), GLUJX5
(512+8 xxx4x
[Conv(3.1), BN, GLU, Conv(3.1), BN]x3
16x 128x128
16x128x 128
Conv(3 BN, GLU
16 12x128x122 128
64x 128x
Go
Conv(3.1), Sigm
64x x28 128
1x 128 128
Conv(3.1), BN, GLU
(16e (e))x
128
G
[Conv(3,1), BN, GLU, Conv(3.1), BN]x2
16x 128 128
16x 128 128
Conv(3.1), BN, GLU
16x 128 x 128
16x 128 x.8x112 128
Comv(3.1), Tanh
128
128
Table 4: Architectural description of the generator of C3-GAN.
Module
Layers
Input size
Output size
Conv(4.2), LReLU
3x 128 x 128
64x64x64
Dbase
[Conv(4.2), BN, [ReLU]x4
64 x4xx466
512x4x4
Conv(3.1), BN, LReLU
512x4x4
512x4x4
Conv(4.4), Squeeze
512x4x4
(t)
Conv(3.1), LReLU
512x4x4
512x4x4
Conv(4.4), Squeeze
$12x4x4
512 (h)
Ve
Linear
Y
(1)
Table Architecture description of the discriminator C3-GAN.
Conv(a,b) 2D Convolution layer with kernel size and stride b
GLU: Gated Linear Units
13

---

Published as a conference paper at ICLR 2022
A.2 ADDITIONAL ABLATION STUDY
Random affine transformation. We experimented with two types of purtubing policy, a weak and strong random affine transformation. The detailed value ranges of their parameters presented in Table 6 The type of perturbation used for each dataset was determined by manually checking randomly perturbed real images. The strong perturbation was only used for CUB dataset, and the rest datasets used the weak perturbation.
Criteria
Scale
Rotation
Translation
Dataset
Wcak perturbation
(0.9, 1.1)
(-2, 2)
(-0.08, 0,08)
Stanford Cars, Stanford Dogs,
Oxford Flower
Strong perturbation
(0.8, 1.5)
(-15, 15)
(-0.15, 0,15)
CUB
Table 6: Details of random affine transformation.
Overclustering. We use four datasets for the evaluation, including CUB, Standford Cars, Stanford Dogs and Oxford Flower. As we described in [.2] we employ the overclustering policy which is to set the number of clusters to be multiple times of the actual number of classes. To find the optimal setting, we compared the results of C3-GAN by setting the number of clusters as 1, 2, and 3 times the actual number of classes. The cluster numbers for each dataset were determined based on the results in Table 7 We found that the performance generally tends to improve the number of clusters increases. This implies that overclustering is indeed helpful for the expressive feature learning. To further investigate the results when additionally conducted all experiments by setting the number clusters as 100 (underclustering) 500 (overclustering). The results represented in Table For the underclustering setting, only report the scores of NMI, since some classes have no matching cluster indices. Despite the less promising performance of the underclustering case, this result implies that method can be applied to any datasets whose cluster size is not available, if we set large enough number of clusters.
not aware of the actual number of classes,
Acc t
NMI
FID
CUB
X
22.7
0.50
18.75
x.2
27.6
0.53
19.37
x3
26.3
0.51
17.13
Stanford Cars
8.3
0.33
64.55
x 2
11.5
0.39
66.65
x
14.1
0.41
67.36
Stanford Dogs
11.8
0.30
$1.37
x 2
15.8
0.35
54.82
x3
17.9
0.36
45.40
Oxford Flower
55.6
0.72
75.12
x
61.7
0.67
74.59
x.
67.8
0.67
64.19
Table 7: Results of hyperparameter search for overclustering.
Acc
NMI t
# of clusters
Bird
Car
Dog
Flower
Bird
Car
Dog
Flower
Actual number of classes x 3 (Overclustering)
26.3 14.1 17.9 67.8
0.51 0.41 0.36 0.67
100 (Underclustering)
0.38 0.25 0.32 0.75
(Overclustering)
23.7 12.1 20.5 70.8 0.51 0.40 0.39 0.65
Table 8: Clustering performance when the number of clusters are arbitrarily set.
14

---

Published as a conference paper at ICLR 2022
A.3 QUALITATIVE ANALYSIS OF CLUSTERING RESULTS
We present the clustered images of CUB and Oxford Flowers datasets, along with the results of MixNMatch and SCAN in Figs. [ 7 From these results, at clustering compared to the baseline methods. It is worth noting that even incorrectly assigned images look very similar with the given bird species for C3-GAN, while the results of other methods contain objects that are quite deviated from the condition. We could observe the similar trend for Oxford Flowers dataset, is shown Fig.
assume that our method is better
(a) C3-GAN (Ours)
(b) SCAN Van Gansboke al. 2020
(c) MixNMatch Li al. 2020
Figure 6: Examples of clustered images of CUB dataset. The cluster indices for the first and the second sets are the ones mapped from the real classes Cerulean Warbler and le Conte Sparrow via Hungarian Algorithm. Red boxes denote incorrectly assigned images.
15

---

Published as a conference paper at ICLR 2022
(a) C3-GAN (Ours)
(b) SCAN Van Gansboke al. 2020
(c) MixNMatch Li
2020
Figure 7: Examples of clustered images of Flower dataset. The cluster indices for the first and the second sets are the ones mapped from the real classes 35 and 66 via Hungarian Algorithm. Red boxes denote incorrectly assigned images.
16

---

Published as a conference paper at ICLR 2022
A.4
ADDITIONAL IMAGE GENERATION RESULTS
A.4.1
CUB
Real
Fake images
Real
Fake images
Real
Fake images
Background
Background
Background
Foroground
Foreground
Foreground
Fixed and variable
Varible and fixed
A.4.2
STANFORD CARS
Real
Fake images
Real
Fake images
Real
Fake images
Background
Background
Background
Foreground
Foreground
Foreground
Fixed and variable
Varible and fixed
17

---

Published as a conference paper at ICLR 2022
A.4.3
STANFORD DOGS
Real
Fake images
Real
Fake images
Real
Fake images
Background
Background
Background
Foreground
Foreground
Foreground
Fixed and variable
Varible c and fixed
A.4.4
OXFORD FLOWER
Fake images
Real
Fake images
Real
Fake images
Background
Background
Background
Foreground
Foreground
Foreground
Fixed and variable
Varible and fixed
18


Reviews:
[{"review_id": "vFPMpBh9us2", "reviewer": "Reviewer_imaQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n1. The method is straightforward and the paper is easy to follow. \n2. The performance boost compared to SOTA for clustering seems to be very good (Table 1). \n3. The ablation study covers major components in the pipeline so it is easy to understand their contributions.\n\n[Weaknesses]\nWhile the paper is focusing on fine-grained image synthesis/clustering, it is not specifically designed for fine-grained classes: the whole pipeline is pretty generic. This is unlike for example the FineGAN paper where they design separate stages for parent and child classes. Other than showing in the experiment that the proposed method just *works* in the fine-grained setting, I'd be interested in seeing more analysis on the advantage of the proposed pipeline in fine-grained setting vs. coarse-grained setting. For example, the paper could show the performance of parent and child classes in those datasets and compare against SOTA. If the proposed method has a similar performance as others in the coarse setting but a clear advantage in the fine-grained setting, that will be a very strong signal.\n\nIn addition, there are some parts in the paper that are unclear or confusing.\n1. Section 3.2, second paragraph. \"The discriminator D aims to learn adversarial features r' --> what does 'adversarial features' mean here? I checked the architecture it seems to be just the discriminator score (a scalar). \n2. What is the effect of under-clustering where Y is much smaller? \n3. The loss defined in Eq. 5 does not use the notion Q or q_k defined in Eq. 4. It is not clear from the text directly what their connection is. It should be written clearly L = -log q_k for example.\n4. Table 1 last sentence: I think OneGAN is an unsupervised method.\n\n[Other suggestions for editing]\n1. The first sentence in the Abstract has redundant words: 'Unsupervisedly' and 'without ground truth annotation' are the same thing.\n2. Figure 4 (a): the same cluster c is very confusing here because those birds are from different classes. I'd suggest simply dropping the symbol c.\n", "summary_of_the_paper": "This paper studies the problem of fine-grained image clustering. Similar to recent work such as OneGAN and FineGAN, the paper proposes to use a GAN setup, called C-3GAN, where the fine-grained image synthesis and clustering are performed in the same end-to-end pipeline. The main contribution is to use a contrastive loss in maximizing the mutual information of the discriminator encoded features and class-centric features. There are also some modifications in the foreground and background synthesis mechanisms compared to prior work such as the FineGAN. Experiments are conducted on fine-grained datasets such as CUB-200-2011.", "summary_of_the_review": "Overall, this paper proposes a straightforward pipeline to synthesize and cluster images in fine-grained classes. A contrastive loss is used in place of the regular softmax loss in the InfoGAN framework. The performance seems to be very solid compared to SOTA methods on both synthesis and clustering tasks. There are some unclear or confusing parts in the paper, but I think given the simplicity and good performance of the method, it might be worth being seen by the community to inspire similar works. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "m9rAeKs1zz5", "reviewer": "Reviewer_5wFE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\t- The problem is known to be valid and challenging\n\t- The method idea, formulations, and figures are clear in general\n\t- The level of novelty is reasonable\n\t- The model is fully unsupervised\nThe authors provide an extensive evaluation and the results are impressive\n\nWeaknesses:\n\t- The part of the contrastive loss is not totally clear. The authors should provide a better intuition of why the contrastive loss improves the feature representation. For example, how are image-latent pairs defined as positive?\n\t- The method focuses on learning cluster granularity for the object only, and not for the background.\n\t- It's unclear why the transformation matrix is used (other than the fact that it's part of PerturbGAN's pipeline)\n\t\nA few comments on the text:\n\t- The phrase \"coarse-grained images\" is inaccurate, the \"coarse-grained\" adjective should refer to the clustering and not the images (in the intro).\n\t- The authors should share more details about the auxiliary distribution mentioned in the abstract and the intro.\n\t- Overall proofreading is required.\nIt would be great to add some of the model's notations to figure 2 (e.g. D_base, psi_r, psi_h)", "summary_of_the_paper": "-- The authors propose C3-GAN, a method that learns \"clustering-friendly\" feature representation for fine-grained clustering (main goal), by learning features of cluster centroids (latent codes) using contrastive loss on the mutual information of image-latent code pairs.\n\n-- The method should improve the GAN's performance in terms of image diversity.\n\n-- The method is unsupervised and applicable for single-object images only.\n\n-- The method is built upon FineGAN (and InfoGAN idea), after adding significant improvements such as removing the dependency on bounding-box labels, applying the mutual information in the embedding space, and directly learning cluster centroids.\n", "summary_of_the_review": "I tend to vote for accepting this paper as I think it proposes a great approach and presents a convincing comparative performance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a0FvrPRXdo", "reviewer": "Reviewer_9p3B", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n\tThe experiments are well executed, and clearly communicated.\n\tThe quantitative results for clustering, and the qualitative figures in the supplementary material are both impressive, and they support the authors' claims\n\tThe method has novel components which are provably attributed to these results.\n\nWeaknesses\n\n\tThe paper is well rounded, with all aspects detailed to completion. There are no obvious weaknesses", "summary_of_the_paper": "The authors undertake the more difficult task of data clustering, based upon fine-grained features. They use contrastive learning for this, in conjunction with GAN losses. Their representation can be used in the downstream task of image generation, where they use their representations' strengths to show improved resilience to mode collapse, while displaying better intra-cluster variance.  ", "summary_of_the_review": "The paper combines several techniques to achieve unsupervised fine-grained clustering of semantic classes. Their representations' high quality is able to drive GAN generation without mode collapse, thereby achieving great two-fold contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7paYF53jrpS", "reviewer": "Reviewer_odf8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n+ Information theory based regularization with contrastive loss learns the features of the cluster centroids. The proposed method is well adapted for this purpose using InfoGAN and PerturbGAN. \n+ The proposed method effectively avoids mode collapse while training generators, and are able to generate, with good control, various objects with varying background (while keeping foreground fixed), and varying foreground objects (while keeping background fixed) better than related methods.\n+ The paper demonstrates that overclustering is a viable approach to hyper-parameter optimization that can achieve better fine-grained clustering due to learning dense features. \n\n*Weaknesses*\n+ The approach is heavily inspired by FineGAN and PerturbGAN. \n+ In overclustering, it is not clear what would happen if the chosen cluster size is such that it is not a multiple of no. of classes? We assume that the data does not have labels in unsupervised feature learning, therefore it would be interesting to see if the cluster size is not a multiple.\n\n*Grammatical/Typographical errors*\n+ Page 1, last line: `...we formulate the auxiliary probability...`\n+ Fig. 2 caption: `C-3GAN`.\n+ Page 5, first line: `...In mathematical, ...`", "summary_of_the_paper": "*The fine-grained class clustering is more challenging than coarse-grained due to lower sample representation and large scale- and color- variations between the fine-grained classes. The main goal of the proposed approach is to learn stronger representations in an unsupervised fashion. To this end, the paper proposes C3-GAN which uses the ability of InfoGAN with contrastive learning to learn feature representations that maximize the mutual information between the latent code and its corresponding observation. The proposed approach is able to achieve best performance in comparison to the related works.* \n\n*The results demonstrated quantitatively and qualitatively on 4 different datasets along with ablation study validate the proposed approach. The method is promising since it is unsupervised way of learning cluster centroid for unlabeled data.*", "summary_of_the_review": "*The paper presents a method that is a hybrid of 2 major previous state-of-art methods. Although heavily inspired, the paper proposes solutions to reduce the drawbacks of the previous approaches - which is the highlight of the contribution. Additionally, the biggest take-away is that the method is unsupervised. Given the breadth of the experiments to validate each of the proposed solutions, and substantial ablation experiments to justify each proposal, the paper overall is a good contribution.*", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers liked the paper. The proposed method contains novel ideas of learning feature representation to maixmize the mutral informatio nbetween the latent code and its corresponding observation for fine-grained class clustering. The model seems to successfully avoid mode collapse while training generators and able to generate various object (foregrounds) with varying backgrounds. The foreground and background control ability is an outstanding feature of the paper. Please incorporate the comments of the reviewers in the final version.

BTW, the real score of this paper should be 7.0 as Reviewer 5wFE commented that he/she would raise the score from 5 to 6 but at the time of this meta review, ths core was not raised. So the final score of the paper should be 8/8/6/6.
Decision: Accept (Spotlight)