Year: 2021

Abstract:
Unsupervised fine-grained class clustering is practical yet challenging due to the difficulty of feature representations learning of subtle object details. We introduce C3-GAN, a method that leverages the categorical inference power of InfoGAN with contrastive learning. We aim to learn feature representations that not only maximize the mutual information between the latent code and its observation, but also encourage the data to form distinct cluster boundaries in the embedding space. Our approach is to train the discriminator optimizing the contrastive loss, where the image-latent pairs that maximize the mutual information are considered as positive pairs and the rest as negative pairs, then we utilize it for clustering images. For this formulation, we map the latent code, which is the input of the generator, to the embedding space of the discriminator and let them act as a cluster centroid. In this way, C3-GAN achieved to learn a clustering-friendly embedding space where each cluster is distinctively separated. Experiments show that C3-GAN achieved state-of-the-art clustering performances on four fine-grained benchmark datasets, while also alleviating the mode collapse phenomenon.

text: Unsupervised fine-grained class clustering is practical yet challenging due to the difficulty of feature representations learning of subtle object details. We introduce C3-GAN, a method that leverages the categorical inference power of InfoGAN with contrastive learning. We aim to learn feature representations that not only maximize the mutual information between the latent code and its observation, but also encourage the data to form distinct cluster boundaries in the embedding space. Our approach is to train the discriminator optimizing the contrastive loss, where the image-latent pairs that maximize the mutual information are considered as positive pairs and the rest as negative pairs, then we utilize it for clustering images. For this formulation, we map the latent code, which is the input of the generator, to the embedding space of the discriminator and let them act as a cluster centroid. In this way, C3-GAN achieved to learn a clustering-friendly embedding space where each cluster is distinctively separated. Experiments show that C3-GAN achieved state-of-the-art clustering performances on four fine-grained benchmark datasets, while also alleviating the mode collapse phenomenon.

heading: 1 INTRODUCTION
text: Unsupervised fine-grained class clustering is to group images of very similar objects (Benny & Wolf, 2020). While a line of works based on multiview-based self-supervised learning (SSL) methods (He et al., 2020; Chen et al., 2020; Grill et al., 2020) shows promising results in the conventional coarsegrained class clustering (Van Gansbeke et al., 2020) task, it is difficult for a fine-grained clustering task to benefit from these methods for two reasons. First, though it is more challenging and ambiguous for finding the distinction between fine-grained classes, datasets of the fine-grained object images are difficult to be large-scale. Figure 1 visually represents that finding distinctions between these classes are more difficult compared to the ones of the coarse-grained object classes. Second, the augmentation process of these methods consider the subtle changes in color or shape, that actually play an important role in sorting classes, as noisy factors. Thus, it is required to find an another approach for the fine-grained class clustering task.
Generative adversarial networks (GAN) (Goodfellow et al., 2014) can be a solution as it is required to learn fine details to generate realistic images. The possible starting point can be InfoGAN (Chen et al., 2016), which has achieved unsupervised categorical inference on MNIST dataset (LeCun et al., 2010) by maximizing the mutual information between the latent code and its observation. The only prior knowledge employed was the number of classes and the fact that the data is uniformly distributed over the classes. FineGAN (Singh et al., 2019) extends InfoGAN by integrating the scene decomposition learning method into the framework, and learns three latent codes that all follow the uniform categorical distribution. Each code has a hierarchical relationship with each other, and is sequentially injected to the generator for a background, a super-class object, and a sub-class object image syntheses. FineGAN demonstrates that the learnt codes for foreground image generation can also be used for clustering real images into their fine-grained classes and outperforms conventional coarse-grained class clustering methods. This result implies that extracting only the foreground part is very helpful for the given task. However, FineGAN requires bounding box annotations and additional training of classifiers, which hinders its applicability. Also, the method lacks the ability of learning the distribution of real images due to the mode collapse phenomenon (Higgins et al., 2017).
In this paper, we propose Constrastive fine-grained Class Clustering GAN (C3-GAN) that deals with the aforementioned problems. We first remove the reliance on human-annotated labels by adopting the method inspired by PerturbGAN (Bielski & Favaro, 2019) for unsupervised scene decomposition learning. And, with the intuition that optimal clustering results would be achieved if clusters of image features are uniformly distributed and can be linearly separable from each other (Wang & Isola, 2020), we train the discriminator to optimize the contrastive loss (He et al., 2020), where the image-latent feature pairs that maximize the mutual information are set as positive pairs. In specific, we map the latent code, which is the input of the generator, onto the embedding space of the discriminator and let it act as a cluster centroid that pulls the feature of the image that has generated with this specific value. By setting the relation with other latent codes as negative pairs, the cluster centroids would be pushing each others so that the embedding space to have the characteristic described earlier. The induced embedding space of the discriminator is also helpful for image generation learning, as the generator is required to have the ability to synthesize certain number of object classes with their distinct characteristics. We have conducted experiments on four fine-grained datasets including CUB, Stanford Cars, Stanford Dogs and Oxford Flower, and show that our C3-GAN achieved state-of-the-art clustering performance on all the datasets while also alleviating the mode collapse issue.
Our main contributions are summarized as three folds:
• We propose a novel form of information-theoretic regularization using the contrastive loss. With the proposed formulation, we directly learn the cluster centroids that would lead the data to form distinct cluster boundaries in the embedding space without falling into a degenerated solution.
• By extracting foreground features via scene decomposition learning, our method achieves the state-of-the-art performance on the fine-grained class clustering while not requiring any human-annotated label.
• Our model is capable of generating images with the controllability over the object classes, and also good at alleviating the mode collapse issue of GAN.

heading: 2 RELATED WORK
text: Unsupervised Clustering. Unsupervised clustering methods can be mainly categorized into information theory-based and Expectation–Maximization (EM)-based approaches. The first approach aims to maximize mutual information between groups of original images and augmented images in an end-to-end fashion (Ji et al., 2019; Zhong et al., 2020). To enhance the performance, IIC (Ji et al., 2019) additionally trains the auxiliary branch using an unlabeled large dataset, and DRC (Zhong et al., 2020) optimizes the contrastive loss on logit features for reducing the intra-class feature variation. EM-based methods decouple the cluster assigning process (E-step) and the feature representations
learning process (M-step) to learn more robust feature representations. Specifically, the feature representations are learned by not only fitting the proto clustering results estimated in the expectation step but also optimizing particular pretext tasks. The clusters are inferred by k-means clustering (Xie et al., 2016; Caron et al., 2018; Liu et al., 2020; Li et al., 2021) or training of an additional encoder (Dizaji et al., 2017). However, these methods suffer from the uneven allocations issue of k-means clustering, which could result in a degenerate solution. For this reason, SCAN (Van Gansbeke et al., 2020) proposes a novel objective function that doesn’t require k-means clustering. Based on the observation that K nearest neighbors of each feature belong to the same ground-truth clusters with the high probability, it trains a classifier that assigns identical clusters to all nearest neighbors and significantly improved the clustering performance. Unlike the prior methods where the cluster centroids are not learned in an end-to-end manner or not the subject of interest, C3-GAN jointly learns the cluster centroids with other parts of the model with clear regularizations on their distribution.
Unsupervised Scene Decomposition. Heavy cost of annotating segmentation masks have drew active research efforts on developing unsupervised methods. Some works address the task in the information-theoretic perspective (Ji et al., 2019; Savarese et al., 2021), but a majority of the works are based on GAN. ReDO (Chen et al. (2019)) learns to discern object masks by redrawing a scene, and PerturbGAN (Bielski & Favaro (2019)), that has separate background and foreground generators, triggers scene decomposition learning by randomly perturbing foreground images. Meanwhile, a line of works utilizes pre-trained high quality models such as StyleGAN (Karras et al., 2019; 2020) and BigGAN (Brock et al., 2019). Labels4Free (Abdal et al., 2021) trains a segmentation network on top of the pre-trained StyleGAN utilizing the fact that inputs of each layer of StyleGAN have different degrees of contribution to foreground synthesis. Voynov & Babenko (2020) and Melas-Kyriazi et al. (2021) explore the latent space of pre-trained GANs to find the perturbing directions that can be used to induce segmentation masks. While all these methods aim to infer foreground masks by training a mask predictor with a synthesized dataset, or projecting real images into the latent space of the pre-trained GAN, C3-GAN is more focusing on the semantic representations learning of the foreground object.
Fine-grained Feature Learning. For hierarchical image generation, some works (Singh et al., 2019; Li et al., 2020; Benny & Wolf, 2020) extend InfoGAN with multiple pairs of adversarial networks, and integrate the scene decomposition learning method into the framework. FineGAN (Singh et al., 2019) finds multiple latent codes using information theory, and each code is used for sequential generation of a background, a super-class object, and a sub-class object image. MixNMatch (Li et al., 2020) and OneGAN (Benny & Wolf, 2020) extend FineGAN with multiple encoders to directly infer these latent codes from real images. Since this autoencoder-based structure could evoke degenerate solutions where only one generator is responsible for synthesizing the entire image, MixNMatch conducts adversarial learning on the joint distribution of the latent code and image (Donahue et al., 2017), and OneGAN trains the model in two stages, learning the generators first and the encoders next. All of these works succeeded in generating images in a hierarchical manner, and further suggest that the learned latent codes can be used for clustering real images into their fine-grained classes. However, these works cannot be applied to datasets unlabeled with bounding box annotations, and require a training of additional networks (classifiers). The proposed model in this paper, C3-GAN, can learn the latent code of foreground region in a completely unsupervised way, and utilize the discriminator for inferring the clusters of the given dataset.

heading: 3 METHOD
text: Given a dataset X = {xi}Ni=1 consisting of single object images, we aim to distribute the set X into Y semantically fine-grained clusters in an unsupervised way. Our method, C3-GAN, is built from the GAN, and the clusters are found in the feature space of the discriminator D, H ∈ Rdh . The cluster index for each image xi is inferred from the cosine distance between embedded image feature hi ∈ Rd h
and the set of cluster centroids l ∈ RY×dh , which are the latent codes c mapped onto the space. Specifically, the index of the closest centroid will be assigned to the query image xi. For the robust fine-grained feature learning, C3-GAN decomposes the scene into the background and foreground part, and associate the latent code only with the foreground region. Moreover, it induces the discriminator D to learn the embedding space where each cluster is uniformly distributed with distinctive boundaries, by formulating the information regularization based on the contrastive loss.

heading: 3.1 PRELIMINARIES
text: The proposed C3-GAN is built from FineGAN (Singh et al., 2019), which is in turn based on InfoGAN (Chen et al., 2016). InfoGAN learns to associate a random latent code c with its observation x̂ by optimizing the model to maximize the mutual information of the two, I(c, x̂). The latent code c can take various forms depending on the prior knowledge of the factor to be inferred, and be defined as the uniform categorical distribution when the purpose is to make the categorical inference of the data. FineGAN learns three such latent codes for hierarchical image generation, and each code is used for a background, a super-class object, and a sub-class object image synthesis respectively. To facilitate this decomposition, FineGAN employs multiple pairs of generator and discriminator and trains an auxiliary background classifier using object bounding box labels. It is demonstrated that the latent codes for the foreground image generation can also be utilized for clustering images according to their fine-grained features. The inference is made by training additional classifiers with synthesized images that are pseudo-labeled with their corresponding latent code values.
Our method differs from FineGAN in that, i) it employs the discriminator D for the clustering without requiring an additional training, and ii) it learns only one latent code c that represents the fine-grained class of an foreground object. Specifically, our generator G synthesizes background image x̂bg ∈ R3×H×W , foreground mask m̂ ∈ R1×H×W and foreground appearance t̂ ∈ R3×H×W with two independent random variables, the noise value z and the latent code c. The noise z ∈ Rdz is sampled from the normal distribution N (0, I). The latent code c ∈ RY is represented as 1-hot encoding where the index k such that ck = 1 is sampled from the uniform categorical distribution U(0, Y − 1). The noise value z is used for generating all image components making the layout of the scene, while the latent code c is used only for the foreground synthesis. To add some randomness, we augmented the 1-hot latent code c to a latent variable c′ ∈ Rdc which is sampled from a gaussian distribution N(µc, σc) where values of the mean µc and the diagonal covariance matrix σc are conditioned on the original code c (Zhang et al., 2017). The final image x̂ ∈ R3×H×W is composited by applying the hadamard product to the generated components, as described in Fig. 2.
To achieve fully unsupervised scene decomposition by removing the dependence on the bounding box annotations, we leverage the random affine transformation method inspired by PerturbGAN (Bielski & Favaro, 2019). In specific, scene decomposition is triggered by perturbing foreground components m̂ and t̂ with the affine transformation matrix Tθ right before the final image composition. The parameters θ of the transformation matrix Tθ include rotation angle, scaling factor and translation distance, and they are all randomly sampled from the uniform distributions with predefined value ranges. We also employ the regularization functions that enforce foreground mask m̂ to be more like a hard mask and to take up the reasonable proportions of the scene,
LmaskG = Ehw[−m̂ log(m̂)− (1− m̂) log(1− m̂)] +max(0, 0.1− Ehw[m̂]) + max(0,Ehw[m̂]− 0.9).
(1)
In summation, the generation process is described as:
c′ ∼ N(µc, σc), x̂bg, m̂, t̂ = G(z, c′), m̂′, t̂′ = Tθ(m̂, t̂), x̂ = x̂bg (1− m̂′) + t̂′ m̂′.
(2)

heading: 3.2 CONTRASTIVE FINE-GRAINED CLASS CLUSTERING
text: We assume that the dataset X would be effectively clustered when i) it explicitly forms discernable cluster boundaries in the embedding spaceH and ii) each cluster centroid l ∈ Rdh exhibits distinct semantic features. This is exactly the space that the discriminator of InfoGAN aims to approach when the latent code c is sampled from the uniform categorical distribution U(0, Y − 1) (Chen et al., 2016). However, while jointly optimizing an adversarial loss, the model has a possibility of falling into the mode collapse where a generator G covers only few confident modes (classes) to easily deceive the discriminator D. This is why InfoGAN cannot infer the classes on real images. To handle this problem, we propose a new form of the auxiliary probability Q(c|x) that represents the mutual information between the latent code c and image x.
Let us describe the mathematical definition in details. The discriminator D aims to learn adversarial features r ∈ R for image authenticity discrimination, and image features h ∈ Rdh for optimizing the information-theoretic regularization. The features are encoded from separate branches ψrx and ψ h x , which has split at the end of the base encoder of the discriminator Dbase. The adversarial features r are learned with the hinge loss as shown in the equation below:
r = ψrx(Dbase(x)), r̂ = ψ r x(Dbase(x̂))
LadvD = min(0, 1− r) + min(0, 1 + r̂) , LadvG = −r̂. (3)
Meanwhile, the image features h can be considered as logits in a classification model. In a supervised learning setting where the ground-truth class labels are given, the data mapped into the embedding space gradually forms the cluster boundaries as the training proceeds, like the one in Fig. 1 (b). However, it is difficult to learn such space in the given problem setting simply by maximizing the mutual information between the latent code c and its observation x̂, I(c; x̂), for the aforementioned reasons. Our solution is to formulate the auxiliary probability Q(c|x) based on the contrastive loss. Specifically, we map the latent code c onto the embedding space of the discriminator with simple linear layer ψc, and let it act as a cluster centroid l that pulls the feature of the image h that has generated with this specific value. By setting the relation with other latent codes as negative pairs, the cluster centroids would be pushing each others so that they are distributed with distinct boundaries. The equation of the proposed formulation is as below:
Q(ck = 1|x̂) = qk = exp(sim(ĥ, lk)/τ)∑Y−1 y=0 exp(sim(ĥ, ly)/τ)
s.t. ĥ = ψhx(Dbase(x̂)) , l = ψc(IY ),
(4)
where sim(a, b) is the cosine distance between vectors a and b, y is the index of the cluster, and τ is a temperature of the softmax function. IY denotes the identity matrix with size Y .
With this function, we expect that the image feature h would be mapped closest to k-th cluster, where k is the index of its corresponding latent code c such that ck = 1, while distances to other centroids ly would be far enough so that the value of qk is close to 1. We additionally maximize the mutual information between the latent code c and the masked foreground images x̂fg to further assist the scene decomposition learning. So the information-theoretic objectives are defined as:
Linfo = − log qk, Linfo fg = − log qfgk . (5)
Following the prior works (Ji et al., 2019; Van Gansbeke et al., 2020), we adopted overclustering strategy so that the model can learn more expressive features. We also regularize the prediction on real images p ∈ RY to further reduce the gap between real and fake images. Specifically, we minimize the entropy of p, H(p), to promote each data to be mapped to one cluster with high confidence and the KL divergence between the averaged prediction p̄ and the uniform distribution u to avoid a
degenerated solution where few clusters are overly allocated. The regularizations are as below:
LentropyD = E[H(p)] +DKL(p̄‖u)
s.t. h = ψhx(Dbase(x)) , pk = exp(sim(h, lk)/τ)∑Y−1 y=0 exp(sim(h, ly)/τ) , p̄ = E[p]. (6)
In summary, our C3-GAN is trained optimizing the following objective functions iteratively:
min D,G
LadvD + LadvG + λ0L entropy D + λ1L mask G + λ2Linfo + λ3Linfo
fg
. (7)

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETTING
text: Datasets. We tested our method on 4 datasets which contain single object images. i) CUB (Wah et al., 2011): 5,994 training and 5,794 test images of 200 bird species. ii) Oxford Flower (Nilsback & Zisserman, 2008): 2,040 training and 6,149 test images of 102 flower categories. iii) Stanford Cars (Krause et al., 2013): 8,144 training and 8,041 test images of 196 car models. iv) Stanford Dogs (Khosla et al., 2011): 12,000 training and 8,580 test images of 120 dog species. Since the CUB and Oxford Flower datasets have a small number of training images, all the experiments were conducted using the entire data as in prior works (Singh et al., 2019; Li et al., 2020; Benny & Wolf, 2020).
Implementation Details. The weights of loss terms (λ0, λ1, λ2, λ3) are set as (0.1, 1, 5, 1), and the temperature τ is set as 0.1. The models are trained with Adam optimizer with learning rate of 0.0002 and momentum coefficients of (0.5, 0.999). We described the network specification in Appendix A.1, and the value ranges of parameters for the random affine transformation and the number of clusters Y in Appendix A.2.
Baselines. We first compare C3-GAN to IIC (Ji et al., 2019) in order to emphasize the difference between the coarse-grained and fine-grained class clustering tasks since it achieved decent clustering results with sobel filtered images where color and texture information is discarded. To claim that optimizing the instance discrimination task is not enough for fine-grained feature representations learning, we also compare with the simple baseline of SimCLR+K-Means (Chen et al., 2020). The work that shows remarkable result on the coarse-grained class clustering, SCAN (Van Gansbeke et al., 2020), are also included to evaluate their method that induces consistent predictions between nearest neighbors on the fine-grained class clustering task. In addition to InfoGAN (Chen et al., 2016), the reported results of FineGAN (Singh et al., 2019), MixNMatch (Li et al., 2020) and OneGAN (Benny & Wolf, 2020) were compared to claim the efficacy of our proposed formulation of the auxiliary probability based on the contrastive loss. For the fair comparison, we mainly compared with the results of the unsupervised version of FineGAN and MixNMatch, which are trained by pseudo-labeling the edges of real images as background regions.

heading: 4.2 FINE-GRAINED CLASS CLUSTERING RESULTS
text: Quantitative results. We evaluate the clustering performance with two metrics: Accuracy (Acc) and Normalized Mutual Information (NMI). The accuracy is calculated based on the optimal mapping between the cluster indices and the real classes inferred from the Hungarian algorithm (Kuhn & Yaw, 1955). The results are presented in Table 1. As shown in Table 1, our C3-GAN outperforms other methods with remarkable margins in terms of Acc on all datasets. For NMI, our method presents better or comparable performances as well. The lowest results of IIC underlines that understanding only structural characteristic is not enough for the fine-grained class clustering task. It is difficult to decide which of the GAN-based methods or the SSL-based methods are better, since both approaches have similar range of scores on every metric and display globally entangled feature spaces as shown in Fig. 3 (a) and (b). From the results of SCAN (2nd rank) and MixNMatch (3rd rank), we conjecture that circumventing the problem by requiring local disentanglement of the embedding space and extracting foreground features via scene decomposition are helpful for improving the fine-grained

heading: Fully Unsupervised Setting
text: clustering performance. The state-of-the-art scores of C3-GAN are not only benefited from the scene decomposition learning, but also boosted from the proposed formulation of Q(c|x) that enforces the clusters to be distinctively distributed in the embedding space as visualized in Fig. 3 (c). Furthermore, we present multiple qualitative results on fine-grained clustering tasks in the Appendix A.3, which show the superiority of our method compared to SCAN and MixNMatch.
Ablation Study. We performed ablation study regarding three factors: i) Overclustering, ii) Perturbing foreground images via random affine transformation, and iii) Our formulation of the auxiliary probability Q(c|x). We investigate the clustering performance on CUB dataset by removing each factor and the results are shown in Table 2. The largest performance degradation is induced by changing the formulation of the auxiliary probability Q(c|x) to a trivial softmax function. This result implies that the high degree of cluster separation, which achieved with our proposed method, is strongly correlated with the clustering performance (Wang & Isola, 2020). We observe that C3-GAN fails to decompose background and foreground parts when the random perturbation Tθ is not applied. The significant performance degradation suggests that the method of extracting only the foreground features has a non-negligible amount of benefit in the clustering performance. Overclustering is also helpful for boosting the performance by allowing a larger capacity for the feature representations learning. We added more ablation results on overclustering in the Appendix A.2.
Acc ↑ NMI ↑
C3-GAN (Ours) 27.6 0.53 i) – Overclustering 22.7 0.50 ii) – Perturbing foreground images via random affine transformation 16.6 0.47 iii) – Contrastive loss based auxiliary probability Q(c|x) 14.5 0.45

heading: 4.3 IMAGE GENERATION RESULTS
text: Qualitative Results. Fig. 4 presents images synthesized by C3-GAN trained on CUB dataset. First, we plot the synthesized novel scenes with the predicted cluster indices of real images in Fig. 4(a). Considering the identical characteristics of synthesized foreground objects, C3-GAN achieved to learn cluster centroids with distinctive fine-grained features set so that it can also clearly mapped with real images. To more investigate the characteristics of two input variables, c and z, we plot the results synthesized by fixing one of the two while diversifying the other one in Fig. 4(b). When c is fixed, all the generated images contain the same bird species while the pose of the bird and backgrounds vary due to the variation of z value. On the contrary, for the variable c and fixed z setting, we could observe images of various bird species with identical pose and background features. We also plot the results of other datasets in the appendix A.4.
We observe that C3-GAN also effectively alleviates the mode collapse issue from Fig. 5. The images are synthesized with the cluster index corresponding to the class of real image. While C3-GAN presents intra-class color, shape, and layout variation for a given flower class, the results of other baselines differ only in layout. We conjecture that this is because the learnt embedding space of the discriminator represents the real data distribution so well, that the generator gets a stronger signal for learning to synthesize realistic images in addition to the adversarial loss. Through the qualitative
investigation of synthesized images, we also found that C3-GAN was the only one that succeeded in scene decomposition learning in an unsupervised way. FineGAN and MixNMatch trained without bounding box annotations fell into a degenerate solution where the entire image is synthesized from the background generator.
Quantitative Results. We evaluate image synthesis performance based on the scores of Frechet Inception Distance (FID) and Inception Score (IS), in addition to the score of reverse KL which measures the distance between the predicted class distribution of real and synthesized images. We compared C3-GAN with the GAN-based models trained in a fully unsupervised setting. The scores of IS and reverse KL were measured with Inception networks fine-tuned on each fine-grained dataset using the method of Cui et al. (2018)1, and the results are displayed in Table 3. C3-GAN presents state-of-the-art or comparable performance in terms of FID and Reverse KL. With the qualitative results together, these results imply our C3-GAN has competitive capability for fine-grained object image generation.

heading: 5 CONCLUSION
text: In this study, we proposed a new GAN-based method, C3-GAN, for unsupervised fine-grained class clustering which is more challenging and less explored than coarse-grained clustering tasks. To improve fine-grained class clustering performance, we formulate the information-theoretic regularization based on the contrastive loss. Also, a scene decomposition-based approach is incorporated into the framework to enforce the model to learn representation focusing on the details of foreground images. We further enhance the performance by adopting the overclustering approach. Extensive experiments show our C3-GAN not only outperforms previous fine-grained clustering methods but also synthesizes fine-grained object images with comparable quality while reducing mode collapse that previous state-of-the-art GAN methods suffer from. Since our main goal is unsupervised finegrained clustering rather than image generation, C3-GAN still has room to be improved despite its comparable results.
1The accuracy of Inception Networks fine-tuned on the CUB/Oxford Flower/Stanford Dogs/Stanford Cars datasets are 0.87/0.94/0.86/0.47 respectively.

heading: ETHICS STATEMENT
text: Remarkable advancement of generative models has played a crucial role as AI tools for text (Brown et al., 2020; Kim et al., 2021), image (Karras et al., 2020; Choi et al., 2020b), audio (Sisman et al., 2020), and multimodal content generation (Tian et al., 2021; Yan et al., 2021). However, as its side effect, many malicious applications have been reported as well, thus leading to severe societal problems, such as deepfake (Dolhansky et al., 2020), fake news (Lazer et al., 2018), and generation biased by training data (Gupta et al., 2021). Our work might be an extension of the harmful effects of generative models. On the contrary, generative models can be a solution to alleviate these side effects via data augmentation (Yang et al., 2019; Choi et al., 2020a). In particular, our method can contribute to alleviating data bias and fine-grained class imbalance. Therefore, many endless efforts are required to make these powerful generative models benefit humans.

heading: REPRODUCIBILITY STATEMENT
text: We described the datasets and baselines compared with our method in detail in 4.1. The details of our model architecture and hyperparameter setup are also explained in A.1 and A.2. And we plan to open the source code of C3-GAN for reproducibility.

heading: A APPENDIX

heading: A.1 NETWORK ARCHITECTURE
text: We present the detailed description of architecture of the generator and the discriminator for our C3-GAN in Table 4 and Table 5, respectively. The colored dots (•, •, •) indicate that outputs of the earlier layers are used as inputs for the later layers. The training is done with 2 NVIDIA-V100 GPUs, and we optimized the model until the convergence of the FID score.
• Conv(a,b) : 2D Convolution layer with kernel size a and stride b • GLU : Gated Linear Units

heading: A.2 ADDITIONAL ABLATION STUDY
text: Random affine transformation. To validate the effects of random affine transformation, we experimented with two types of purtubing policies such as weak and strong perturbation. Their detailed value ranges are presented in Table 6. The type of perturbation to use for each dataset was determined by manually checking the results of randomly perturbed real images. The strong perturbation was used for CUB dataset, and the rest datasets were trained with the weak perturbation.
Cluster numbers for each dataset. We use four datasets for evaluation, including CUB, Oxford Flower, Standford Cars, and Stanford Dogs. As depicted in 3.2, we employ the overclustering approaches to improve the performances, which is to set the cluster number to be multiple times as the actual number of classes of the dataset. For finding optimal setting, we compared the results of C3-GAN by setting the number of clusters as 1, 2, and 3 times the actual number of classes. We found that performance tends to improve as the number of clusters increases. This implies that overclustering is indeed helpful for expressive feature learning. The cluster numbers for each dataset were determined based on the results in Table 7. To further investigate the results when we are not aware of the actual number of classes, we additionally conducted all experiments by setting the number of clusters as 100 (underclustering) or 500 (overclustering). The results are represented in Table 8. For the underclustering setting, we only report the scores of NMI, since some classes have no matching cluster indices. Despite the less promising results of the underclustering, this implies that our method can be applied to datasets whose cluster size is not available if we set a large enough number of clusters.

heading: A.3 QUALITATIVE ANALYSIS OF CLUSTERING RESULTS
text: We present the clustered images by our C3-GAN on two classes for CUB and Oxford Flowers datasets, comparing the results of MixNMatch and SCAN in Figs. 6 and 7. From Fig. 6, our method presents better clustered results outperforming two comparable methods. It is worth noting that even the objects in images incorrectly clustered by C3-GAN look very similar to given object class while the cluster results of other two methods contain images of objects that seldom resemble the given class object. We can confirm the consistent results on the Flower dataset as shown in Fig. 7.

heading: A.4 ADDITIONAL IMAGE GENERATION RESULTS
text: A.4.1 CUB
Real Fake images Real Fake images Real Fake images
Background Background Background
+ Foreground + Foreground + Foreground
Fixed c and variable z Varible c and fixed z

heading: A.4.2 STANFORD CARS
text: Real Fake images Real Fake images Real Fake images
Background Background Background
+ Foreground + Foreground + Foreground
Fixed c and variable z Varible c and fixed z

heading: A.4.3 STANFORD DOGS
text: Real Fake images Real Fake images Real Fake images
Background Background Background
+ Foreground + Foreground + Foreground
Fixed c and variable z Varible c and fixed z

heading: A.4.4 OXFORD FLOWER
text: Real Fake images Real Fake images Real Fake images
Background Background Background
+ Foreground + Foreground + Foreground
Fixed c and variable z Varible c and fixed z

Review summaries:
-- The authors propose C3-GAN, a method that learns "clustering-friendly" feature representation for fine-grained clustering (main goal), by learning features of cluster centroids (latent codes) using contrastive loss on the mutual information of image-latent code pairs.  -- The method should improve the GAN's performance in terms of image diversity.  -- The method is unsupervised and applicable for single-object images only.  -- The method is built upon FineGAN (and InfoGAN idea), after adding significant improvements such as removing the dependency on bounding-box labels, applying the mutual information in the embedding space, and directly learning cluster centroids. 

*The fine-grained class clustering is more challenging than coarse-grained due to lower sample representation and large scale- and color- variations between the fine-grained classes. The main goal of the proposed approach is to learn stronger representations in an unsupervised fashion. To this end, the paper proposes C3-GAN which uses the ability of InfoGAN with contrastive learning to learn feature representations that maximize the mutual information between the latent code and its corresponding observation. The proposed approach is able to achieve best performance in comparison to the related works.*   *The results demonstrated quantitatively and qualitatively on 4 different datasets along with ablation study validate the proposed approach. The method is promising since it is unsupervised way of learning cluster centroid for unlabeled data.*

This paper studies the problem of fine-grained image clustering. Similar to recent work such as OneGAN and FineGAN, the paper proposes to use a GAN setup, called C-3GAN, where the fine-grained image synthesis and clustering are performed in the same end-to-end pipeline. The main contribution is to use a contrastive loss in maximizing the mutual information of the discriminator encoded features and class-centric features. There are also some modifications in the foreground and background synthesis mechanisms compared to prior work such as the FineGAN. Experiments are conducted on fine-grained datasets such as CUB-200-2011.

The authors undertake the more difficult task of data clustering, based upon fine-grained features. They use contrastive learning for this, in conjunction with GAN losses. Their representation can be used in the downstream task of image generation, where they use their representations' strengths to show improved resilience to mode collapse, while displaying better intra-cluster variance.