Contrastive Fine-grained Class Clustering via Generative Adversarial Networks

Unsupervised fine-grained class clustering is practical yet challenging due to the difficulty of feature representations learning of subtle object details . We introduce C3-GAN , a method that leverages the categorical inference power of InfoGAN with contrastive learning . We aim to learn feature representations that not only maximize the mutual information between the latent code and its observation , but also encourage the data to form distinct cluster boundaries in the embedding space . Our approach is to train the discriminator optimizing the contrastive loss , where the image-latent pairs that maximize the mutual information are considered as positive pairs and the rest as negative pairs , then we utilize it for clustering images . For this formulation , we map the latent code , which is the input of the generator , to the embedding space of the discriminator and let them act as a cluster centroid . In this way , C3-GAN achieved to learn a clustering-friendly embedding space where each cluster is distinctively separated . Experiments show that C3-GAN achieved state-of-the-art clustering performances on four fine-grained benchmark datasets , while also alleviating the mode collapse phenomenon . 1 INTRODUCTION . Unsupervised fine-grained class clustering is to group images of very similar objects ( Benny & Wolf , 2020 ) . While a line of works based on multiview-based self-supervised learning ( SSL ) methods ( He et al. , 2020 ; Chen et al. , 2020 ; Grill et al. , 2020 ) shows promising results in the conventional coarsegrained class clustering ( Van Gansbeke et al. , 2020 ) task , it is difficult for a fine-grained clustering task to benefit from these methods for two reasons . First , though it is more challenging and ambiguous for finding the distinction between fine-grained classes , datasets of the fine-grained object images are difficult to be large-scale . Figure 1 visually represents that finding distinctions between these classes are more difficult compared to the ones of the coarse-grained object classes . Second , the augmentation process of these methods consider the subtle changes in color or shape , that actually play an important role in sorting classes , as noisy factors . Thus , it is required to find an another approach for the fine-grained class clustering task . Generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) can be a solution as it is required to learn fine details to generate realistic images . The possible starting point can be InfoGAN ( Chen et al. , 2016 ) , which has achieved unsupervised categorical inference on MNIST dataset ( LeCun et al. , 2010 ) by maximizing the mutual information between the latent code and its observation . The only prior knowledge employed was the number of classes and the fact that the data is uniformly distributed over the classes . FineGAN ( Singh et al. , 2019 ) extends InfoGAN by integrating the scene decomposition learning method into the framework , and learns three latent codes that all follow the uniform categorical distribution . Each code has a hierarchical relationship with each other , and is sequentially injected to the generator for a background , a super-class object , and a sub-class object image syntheses . FineGAN demonstrates that the learnt codes for foreground image generation can also be used for clustering real images into their fine-grained classes and outperforms conventional coarse-grained class clustering methods . This result implies that extracting only the foreground part is very helpful for the given task . However , FineGAN requires bounding box annotations and additional training of classifiers , which hinders its applicability . Also , the method lacks the ability of learning the distribution of real images due to the mode collapse phenomenon ( Higgins et al. , 2017 ) . In this paper , we propose Constrastive fine-grained Class Clustering GAN ( C3-GAN ) that deals with the aforementioned problems . We first remove the reliance on human-annotated labels by adopting the method inspired by PerturbGAN ( Bielski & Favaro , 2019 ) for unsupervised scene decomposition learning . And , with the intuition that optimal clustering results would be achieved if clusters of image features are uniformly distributed and can be linearly separable from each other ( Wang & Isola , 2020 ) , we train the discriminator to optimize the contrastive loss ( He et al. , 2020 ) , where the image-latent feature pairs that maximize the mutual information are set as positive pairs . In specific , we map the latent code , which is the input of the generator , onto the embedding space of the discriminator and let it act as a cluster centroid that pulls the feature of the image that has generated with this specific value . By setting the relation with other latent codes as negative pairs , the cluster centroids would be pushing each others so that the embedding space to have the characteristic described earlier . The induced embedding space of the discriminator is also helpful for image generation learning , as the generator is required to have the ability to synthesize certain number of object classes with their distinct characteristics . We have conducted experiments on four fine-grained datasets including CUB , Stanford Cars , Stanford Dogs and Oxford Flower , and show that our C3-GAN achieved state-of-the-art clustering performance on all the datasets while also alleviating the mode collapse issue . Our main contributions are summarized as three folds : • We propose a novel form of information-theoretic regularization using the contrastive loss . With the proposed formulation , we directly learn the cluster centroids that would lead the data to form distinct cluster boundaries in the embedding space without falling into a degenerated solution . • By extracting foreground features via scene decomposition learning , our method achieves the state-of-the-art performance on the fine-grained class clustering while not requiring any human-annotated label . • Our model is capable of generating images with the controllability over the object classes , and also good at alleviating the mode collapse issue of GAN . 2 RELATED WORK . Unsupervised Clustering . Unsupervised clustering methods can be mainly categorized into information theory-based and Expectation–Maximization ( EM ) -based approaches . The first approach aims to maximize mutual information between groups of original images and augmented images in an end-to-end fashion ( Ji et al. , 2019 ; Zhong et al. , 2020 ) . To enhance the performance , IIC ( Ji et al. , 2019 ) additionally trains the auxiliary branch using an unlabeled large dataset , and DRC ( Zhong et al. , 2020 ) optimizes the contrastive loss on logit features for reducing the intra-class feature variation . EM-based methods decouple the cluster assigning process ( E-step ) and the feature representations learning process ( M-step ) to learn more robust feature representations . Specifically , the feature representations are learned by not only fitting the proto clustering results estimated in the expectation step but also optimizing particular pretext tasks . The clusters are inferred by k-means clustering ( Xie et al. , 2016 ; Caron et al. , 2018 ; Liu et al. , 2020 ; Li et al. , 2021 ) or training of an additional encoder ( Dizaji et al. , 2017 ) . However , these methods suffer from the uneven allocations issue of k-means clustering , which could result in a degenerate solution . For this reason , SCAN ( Van Gansbeke et al. , 2020 ) proposes a novel objective function that doesn ’ t require k-means clustering . Based on the observation that K nearest neighbors of each feature belong to the same ground-truth clusters with the high probability , it trains a classifier that assigns identical clusters to all nearest neighbors and significantly improved the clustering performance . Unlike the prior methods where the cluster centroids are not learned in an end-to-end manner or not the subject of interest , C3-GAN jointly learns the cluster centroids with other parts of the model with clear regularizations on their distribution . Unsupervised Scene Decomposition . Heavy cost of annotating segmentation masks have drew active research efforts on developing unsupervised methods . Some works address the task in the information-theoretic perspective ( Ji et al. , 2019 ; Savarese et al. , 2021 ) , but a majority of the works are based on GAN . ReDO ( Chen et al . ( 2019 ) ) learns to discern object masks by redrawing a scene , and PerturbGAN ( Bielski & Favaro ( 2019 ) ) , that has separate background and foreground generators , triggers scene decomposition learning by randomly perturbing foreground images . Meanwhile , a line of works utilizes pre-trained high quality models such as StyleGAN ( Karras et al. , 2019 ; 2020 ) and BigGAN ( Brock et al. , 2019 ) . Labels4Free ( Abdal et al. , 2021 ) trains a segmentation network on top of the pre-trained StyleGAN utilizing the fact that inputs of each layer of StyleGAN have different degrees of contribution to foreground synthesis . Voynov & Babenko ( 2020 ) and Melas-Kyriazi et al . ( 2021 ) explore the latent space of pre-trained GANs to find the perturbing directions that can be used to induce segmentation masks . While all these methods aim to infer foreground masks by training a mask predictor with a synthesized dataset , or projecting real images into the latent space of the pre-trained GAN , C3-GAN is more focusing on the semantic representations learning of the foreground object . Fine-grained Feature Learning . For hierarchical image generation , some works ( Singh et al. , 2019 ; Li et al. , 2020 ; Benny & Wolf , 2020 ) extend InfoGAN with multiple pairs of adversarial networks , and integrate the scene decomposition learning method into the framework . FineGAN ( Singh et al. , 2019 ) finds multiple latent codes using information theory , and each code is used for sequential generation of a background , a super-class object , and a sub-class object image . MixNMatch ( Li et al. , 2020 ) and OneGAN ( Benny & Wolf , 2020 ) extend FineGAN with multiple encoders to directly infer these latent codes from real images . Since this autoencoder-based structure could evoke degenerate solutions where only one generator is responsible for synthesizing the entire image , MixNMatch conducts adversarial learning on the joint distribution of the latent code and image ( Donahue et al. , 2017 ) , and OneGAN trains the model in two stages , learning the generators first and the encoders next . All of these works succeeded in generating images in a hierarchical manner , and further suggest that the learned latent codes can be used for clustering real images into their fine-grained classes . However , these works can not be applied to datasets unlabeled with bounding box annotations , and require a training of additional networks ( classifiers ) . The proposed model in this paper , C3-GAN , can learn the latent code of foreground region in a completely unsupervised way , and utilize the discriminator for inferring the clusters of the given dataset . 3 METHOD . Given a dataset X = { xi } Ni=1 consisting of single object images , we aim to distribute the set X into Y semantically fine-grained clusters in an unsupervised way . Our method , C3-GAN , is built from the GAN , and the clusters are found in the feature space of the discriminator D , H ∈ Rdh . The cluster index for each image xi is inferred from the cosine distance between embedded image feature hi ∈ Rd h and the set of cluster centroids l ∈ RY×dh , which are the latent codes c mapped onto the space . Specifically , the index of the closest centroid will be assigned to the query image xi . For the robust fine-grained feature learning , C3-GAN decomposes the scene into the background and foreground part , and associate the latent code only with the foreground region . Moreover , it induces the discriminator D to learn the embedding space where each cluster is uniformly distributed with distinctive boundaries , by formulating the information regularization based on the contrastive loss .

Summary: *The fine-grained class clustering is more challenging than coarse-grained due to lower sample representation and large scale- and color- variations between the fine-grained classes. The main goal of the proposed approach is to learn stronger representations in an unsupervised fashion. To this end, the paper proposes C3-GAN which uses the ability of InfoGAN with contrastive learning to learn feature representations that maximize the mutual information between the latent code and its corresponding observation. The proposed approach is able to achieve best performance in comparison to the related works.*   *The results demonstrated quantitatively and qualitatively on 4 different datasets along with ablation study validate the proposed approach. The method is promising since it is unsupervised way of learning cluster centroid for unlabeled data.*