Contrastive Fine-grained Class Clustering via Generative Adversarial Networks

Unsupervised fine-grained class clustering is practical yet challenging due to the difficulty of feature representations learning of subtle object details . We introduce C3-GAN , a method that leverages the categorical inference power of InfoGAN with contrastive learning . We aim to learn feature representations that not only maximize the mutual information between the latent code and its observation , but also encourage the data to form distinct cluster boundaries in the embedding space . Our approach is to train the discriminator optimizing the contrastive loss , where the image-latent pairs that maximize the mutual information are considered as positive pairs and the rest as negative pairs , then we utilize it for clustering images . For this formulation , we map the latent code , which is the input of the generator , to the embedding space of the discriminator and let them act as a cluster centroid . In this way , C3-GAN achieved to learn a clustering-friendly embedding space where each cluster is distinctively separated . Experiments show that C3-GAN achieved state-of-the-art clustering performances on four fine-grained benchmark datasets , while also alleviating the mode collapse phenomenon . 1 INTRODUCTION . Unsupervised fine-grained class clustering is to group images of very similar objects ( Benny & Wolf , 2020 ) . While a line of works based on multiview-based self-supervised learning ( SSL ) methods ( He et al. , 2020 ; Chen et al. , 2020 ; Grill et al. , 2020 ) shows promising results in the conventional coarsegrained class clustering ( Van Gansbeke et al. , 2020 ) task , it is difficult for a fine-grained clustering task to benefit from these methods for two reasons . First , though it is more challenging and ambiguous for finding the distinction between fine-grained classes , datasets of the fine-grained object images are difficult to be large-scale . Figure 1 visually represents that finding distinctions between these classes are more difficult compared to the ones of the coarse-grained object classes . Second , the augmentation process of these methods consider the subtle changes in color or shape , that actually play an important role in sorting classes , as noisy factors . Thus , it is required to find an another approach for the fine-grained class clustering task . Generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) can be a solution as it is required to learn fine details to generate realistic images . The possible starting point can be InfoGAN ( Chen et al. , 2016 ) , which has achieved unsupervised categorical inference on MNIST dataset ( LeCun et al. , 2010 ) by maximizing the mutual information between the latent code and its observation . The only prior knowledge employed was the number of classes and the fact that the data is uniformly distributed over the classes . FineGAN ( Singh et al. , 2019 ) extends InfoGAN by integrating the scene decomposition learning method into the framework , and learns three latent codes that all follow the uniform categorical distribution . Each code has a hierarchical relationship with each other , and is sequentially injected to the generator for a background , a super-class object , and a sub-class object image syntheses . FineGAN demonstrates that the learnt codes for foreground image generation can also be used for clustering real images into their fine-grained classes and outperforms conventional coarse-grained class clustering methods . This result implies that extracting only the foreground part is very helpful for the given task . However , FineGAN requires bounding box annotations and additional training of classifiers , which hinders its applicability . Also , the method lacks the ability of learning the distribution of real images due to the mode collapse phenomenon ( Higgins et al. , 2017 ) . In this paper , we propose Constrastive fine-grained Class Clustering GAN ( C3-GAN ) that deals with the aforementioned problems . We first remove the reliance on human-annotated labels by adopting the method inspired by PerturbGAN ( Bielski & Favaro , 2019 ) for unsupervised scene decomposition learning . And , with the intuition that optimal clustering results would be achieved if clusters of image features are uniformly distributed and can be linearly separable from each other ( Wang & Isola , 2020 ) , we train the discriminator to optimize the contrastive loss ( He et al. , 2020 ) , where the image-latent feature pairs that maximize the mutual information are set as positive pairs . In specific , we map the latent code , which is the input of the generator , onto the embedding space of the discriminator and let it act as a cluster centroid that pulls the feature of the image that has generated with this specific value . By setting the relation with other latent codes as negative pairs , the cluster centroids would be pushing each others so that the embedding space to have the characteristic described earlier . The induced embedding space of the discriminator is also helpful for image generation learning , as the generator is required to have the ability to synthesize certain number of object classes with their distinct characteristics . We have conducted experiments on four fine-grained datasets including CUB , Stanford Cars , Stanford Dogs and Oxford Flower , and show that our C3-GAN achieved state-of-the-art clustering performance on all the datasets while also alleviating the mode collapse issue . Our main contributions are summarized as three folds : • We propose a novel form of information-theoretic regularization using the contrastive loss . With the proposed formulation , we directly learn the cluster centroids that would lead the data to form distinct cluster boundaries in the embedding space without falling into a degenerated solution . • By extracting foreground features via scene decomposition learning , our method achieves the state-of-the-art performance on the fine-grained class clustering while not requiring any human-annotated label . • Our model is capable of generating images with the controllability over the object classes , and also good at alleviating the mode collapse issue of GAN . 2 RELATED WORK . Unsupervised Clustering . Unsupervised clustering methods can be mainly categorized into information theory-based and Expectation–Maximization ( EM ) -based approaches . The first approach aims to maximize mutual information between groups of original images and augmented images in an end-to-end fashion ( Ji et al. , 2019 ; Zhong et al. , 2020 ) . To enhance the performance , IIC ( Ji et al. , 2019 ) additionally trains the auxiliary branch using an unlabeled large dataset , and DRC ( Zhong et al. , 2020 ) optimizes the contrastive loss on logit features for reducing the intra-class feature variation . EM-based methods decouple the cluster assigning process ( E-step ) and the feature representations learning process ( M-step ) to learn more robust feature representations . Specifically , the feature representations are learned by not only fitting the proto clustering results estimated in the expectation step but also optimizing particular pretext tasks . The clusters are inferred by k-means clustering ( Xie et al. , 2016 ; Caron et al. , 2018 ; Liu et al. , 2020 ; Li et al. , 2021 ) or training of an additional encoder ( Dizaji et al. , 2017 ) . However , these methods suffer from the uneven allocations issue of k-means clustering , which could result in a degenerate solution . For this reason , SCAN ( Van Gansbeke et al. , 2020 ) proposes a novel objective function that doesn ’ t require k-means clustering . Based on the observation that K nearest neighbors of each feature belong to the same ground-truth clusters with the high probability , it trains a classifier that assigns identical clusters to all nearest neighbors and significantly improved the clustering performance . Unlike the prior methods where the cluster centroids are not learned in an end-to-end manner or not the subject of interest , C3-GAN jointly learns the cluster centroids with other parts of the model with clear regularizations on their distribution . Unsupervised Scene Decomposition . Heavy cost of annotating segmentation masks have drew active research efforts on developing unsupervised methods . Some works address the task in the information-theoretic perspective ( Ji et al. , 2019 ; Savarese et al. , 2021 ) , but a majority of the works are based on GAN . ReDO ( Chen et al . ( 2019 ) ) learns to discern object masks by redrawing a scene , and PerturbGAN ( Bielski & Favaro ( 2019 ) ) , that has separate background and foreground generators , triggers scene decomposition learning by randomly perturbing foreground images . Meanwhile , a line of works utilizes pre-trained high quality models such as StyleGAN ( Karras et al. , 2019 ; 2020 ) and BigGAN ( Brock et al. , 2019 ) . Labels4Free ( Abdal et al. , 2021 ) trains a segmentation network on top of the pre-trained StyleGAN utilizing the fact that inputs of each layer of StyleGAN have different degrees of contribution to foreground synthesis . Voynov & Babenko ( 2020 ) and Melas-Kyriazi et al . ( 2021 ) explore the latent space of pre-trained GANs to find the perturbing directions that can be used to induce segmentation masks . While all these methods aim to infer foreground masks by training a mask predictor with a synthesized dataset , or projecting real images into the latent space of the pre-trained GAN , C3-GAN is more focusing on the semantic representations learning of the foreground object . Fine-grained Feature Learning . For hierarchical image generation , some works ( Singh et al. , 2019 ; Li et al. , 2020 ; Benny & Wolf , 2020 ) extend InfoGAN with multiple pairs of adversarial networks , and integrate the scene decomposition learning method into the framework . FineGAN ( Singh et al. , 2019 ) finds multiple latent codes using information theory , and each code is used for sequential generation of a background , a super-class object , and a sub-class object image . MixNMatch ( Li et al. , 2020 ) and OneGAN ( Benny & Wolf , 2020 ) extend FineGAN with multiple encoders to directly infer these latent codes from real images . Since this autoencoder-based structure could evoke degenerate solutions where only one generator is responsible for synthesizing the entire image , MixNMatch conducts adversarial learning on the joint distribution of the latent code and image ( Donahue et al. , 2017 ) , and OneGAN trains the model in two stages , learning the generators first and the encoders next . All of these works succeeded in generating images in a hierarchical manner , and further suggest that the learned latent codes can be used for clustering real images into their fine-grained classes . However , these works can not be applied to datasets unlabeled with bounding box annotations , and require a training of additional networks ( classifiers ) . The proposed model in this paper , C3-GAN , can learn the latent code of foreground region in a completely unsupervised way , and utilize the discriminator for inferring the clusters of the given dataset . 3 METHOD . Given a dataset X = { xi } Ni=1 consisting of single object images , we aim to distribute the set X into Y semantically fine-grained clusters in an unsupervised way . Our method , C3-GAN , is built from the GAN , and the clusters are found in the feature space of the discriminator D , H ∈ Rdh . The cluster index for each image xi is inferred from the cosine distance between embedded image feature hi ∈ Rd h and the set of cluster centroids l ∈ RY×dh , which are the latent codes c mapped onto the space . Specifically , the index of the closest centroid will be assigned to the query image xi . For the robust fine-grained feature learning , C3-GAN decomposes the scene into the background and foreground part , and associate the latent code only with the foreground region . Moreover , it induces the discriminator D to learn the embedding space where each cluster is uniformly distributed with distinctive boundaries , by formulating the information regularization based on the contrastive loss .

Summary: -- The authors propose C3-GAN, a method that learns "clustering-friendly" feature representation for fine-grained clustering (main goal), by learning features of cluster centroids (latent codes) using contrastive loss on the mutual information of image-latent code pairs.  -- The method should improve the GAN's performance in terms of image diversity.  -- The method is unsupervised and applicable for single-object images only.  -- The method is built upon FineGAN (and InfoGAN idea), after adding significant improvements such as removing the dependency on bounding-box labels, applying the mutual information in the embedding space, and directly learning cluster centroids.