Exploring Non-Contrastive Representation Learning for Deep Clustering

1 INTRODUCTION . Deep clustering is gaining considerable attention as it can learn representation of images and perform clustering in an end-to-end fashion . Remarkably , contrastive learning-based methods ( Wang et al. , 2021 ; Van Gansbeke et al. , 2020 ; Li et al. , 2021a ; b ; Tao et al. , 2021 ; Tsai et al. , 2021 ; Niu et al. , 2021 ) have become the main thrust to advance the representation of images on several complex benchmark datasets , significantly contributing to the clustering performance . In addition , some contrastive learning methods such as MoCo ( He et al. , 2020 ) and SimCLR ( Chen et al. , 2020 ) usually require specially designed losses ( Wang et al. , 2021 ; Li et al. , 2021a ; b ; Tao et al. , 2021 ; Tsai et al. , 2021 ) or an extra pre-training stage for more discriminative representations ( Van Gansbeke et al. , 2020 ; Niu et al. , 2021 ) . Although achieving promising clustering results , contrastive learning requires a large number of negative examples to achieve the instance-wise discrimination in an embedding space where all instances are well-separated . The constructed negative pairs usually require a large batch size ( Chen et al. , 2020 ) , memory queue ( He et al. , 2020 ) , or memory bank ( Wu et al. , 2018 ) , which not only bring extra computational cost but also give rise to class collision issue ( Saunshi et al. , 2019 ) . Here , the class collision issue refers to that different instances from the same semantic class are regarded as negative pairs , hurting the representation learning for clustering . A question naturally arises : are negative examples necessary for deep clustering ? Another kind of self-supervised learning is the non-contrastive methods such as BYOL ( Grill et al. , 2020 ) and SimSiam ( Chen & He , 2021 ) , which use the representations of one augmented view to predict another view . Their success demonstrates that negative examples are not the key to avoiding representation collapse . However , to the best of our knowledge , almost all recent successful literature of deep clustering is built upon contrastive learning-based methods such as MoCo ( He et al. , 2020 ) and SimCLR ( Chen et al. , 2020 ) . There is a general consensus that the negative examples are helpful to stabilize the training of representation learning for deep clustering . As discussed in ( Wang & Isola , 2020 ) , the typical contrastive loss can be identified into two properties : 1 ) alignment term to improve the closeness of positive pairs ; and 2 ) uniformity term to encourage instances to be uniformly distributed on a unit hypersphere . In contrast , non-contrastive methods such as BYOL only optimize the alignment term , leading to unstable training and suffering from the representation collapse—which may be worsen when adding extra losses . To tackle the class collision issue , we explore the non-contrastive representation learning for deep clustering , termed non-contrastive clustering or NCC , which is based on BYOL , a representative method without negative examples . First , instead of negative sampling that is a double-edged sword , i.e. , causing class collision issue but improving the training stability , we propose to align one augmented view of the instance with the neighbors of another view in the embedding space , called positive sampling strategy , which can avoid the class collision issue and hence improve the withincluster compactness . Second , as for the clustering task , the different clusters are truly negative pairs for contrastive loss . To this end , we propose to encourage the alignment between two augmented views of prototypes and the uniformity among all prototypes , named prototypical contrastive loss or ProtoCL , which can maximize the inter-cluster distance . Moreover , we formulate our method into an EM framework , in which we iteratively perform E-step as estimating the pseudo-labels of instances and distribution of prototypes via spherical k-means based on the target network and Mstep as optimizing the online network via the proposed losses . As a result , NCC is able to form an embedding space where all clusters are well-separated and within-cluster examples are compact . The contributions of this paper are summarized as follows : • We explore the non-contrastive representation learning for deep clustering by proposing noncontrastive clustering or NCC , which is based on the Bootstrap Your Own Latent ( BYOL ) , a representative method without negative examples . • We propose a positive sampling strategy to augment instance alignment by taking into account neighboring positive examples in the embedding space , which can avoid the class collision issue and hence improve the within-cluster compactness . • We propose a novel prototypical contrastive loss or ProtoCL , which can align one augmented view of prototypes with another view and encourage the uniformity among all prototypes on a unit hypersphere , hence maximizing the inter-cluster distance . • We formulate our method into an EM framework , in which we can iteratively estimate the pseudolabels and distribution of prototypes via spherical k-means based on the target network and optimize the online network via the proposed losses . • Extensively experimental results on several benchmark datasets as well as ImageNet-1K demonstrate that NCC outperforms the existing state-of-the-art methods by a significant margin . 2 RELATED WORK . Deep clustering can be significantly advanced by discriminative representations . Examples of traditional deep clustering methods include : Xie et al . ( 2016 ) ; Yang et al . ( 2017 ) use autoencoders to simultaneously perform representation learning and clustering ; Chang et al . ( 2017 ) ; Haeusser et al . ( 2018 ) ; Wu et al . ( 2019 ) ; Ji et al . ( 2019 ) learn pair-wise relationships between original and augmented instances . However , they often suffer from inferior performance on some complex datasets such as CIFAR-20 . Inspired by the success of contrastive learning , recent studies turn to exploit the discriminative representations learned from contrastive learning to assist the downstream clustering tasks ( Van Gansbeke et al. , 2020 ; Niu et al. , 2021 ) or simultaneously optimize representation learning and clustering ( Tao et al. , 2021 ; Tsai et al. , 2021 ; Li et al. , 2021a ; Shen et al. , 2021 ) . SCAN ( Van Gansbeke et al. , 2020 ) uses the model pre-trained by SimCLR to yield the confident pseudo-labels . IDFD ( Tao et al. , 2021 ) proposes to perform both instance discrimination and feature decorrelation . GCC ( Zhong et al. , 2021 ) and WCL ( Zheng et al. , 2021 ) build a graph to label the neighbor samples as pseudo-positive examples , however , they still suffer from the class collision issue due to the contrastive loss involved and these pseudo-positive examples that may not be truly positive . All of them are built upon the contrastive learning framework , which means that they require a large number of negative examples for training stability , inevitably giving rise to class collision issue . Different from prior work , this paper explores the non-contrastive self-supervised methods , i.e. , BYOL , to achieve both representation learning and clustering . We note that Regatti et al . ( 2021 ) ; Lee et al . ( 2020 ) have tried to build the clustering framework based on BYOL , however , their methods do not consider improving within-cluster compactness and maximizing inter-cluster distance like ours . Therefore , to the best of our knowledge , this is the first successful attempt that introduces the non-contrastive representation learning into deep clustering that yields a substantial performance improvement over previous state-of-the-art methods . In Appendix A , we present related work on self-supervised learning and difference from existing methods including CC ( Li et al. , 2021b ) , GCC ( Zhong et al. , 2021 ) , WCL ( Zheng et al. , 2021 ) , and PCL ( Li et al. , 2021a ) . 3 PRELIMINARY . The most successful self-supervised learning methods in recent years can be roughly divided into contrastive ( Chen et al. , 2020 ; He et al. , 2020 ) and non-contrastive ( Grill et al. , 2020 ; Chen & He , 2021 ) . Here , we briefly summarize their formulas and discuss their difference . Contrastive learning . Contrastive learning methods perform instance-wise discrimination ( Wu et al. , 2018 ) using the InfoNCE loss ( Oord et al. , 2018 ) . Formally , assume that we have one instance x , its augmented version x+ by using random data augmentation , and a set of M negative examples drawn from the dataset , { x−1 , x − 2 , . . . , x − M } . The contrastive learning aims to learn an embedding function f that maps x onto a unit hypersphere , in which the InfoNCE loss can be defined as : Lcontr = − log exp ( f ( x ) Tf ( x+ ) /τ ) exp ( f ( x ) Tf ( x+ ) /τ ) + ∑M i=1 exp ( f ( x ) Tf ( x−i ) /τ ) ( 1 ) ≈ −f ( x ) Tf ( x+ ) /τ + log ∑M i=1 exp ( f ( x ) Tf ( x−i ) /τ ) , ( 2 ) where the first and second terms in Eq . ( 2 ) refer to as instance alignment and instance uniformity , respectively . Here , we assume that the output of f ( · ) is ` 2 normalized . That is , the representation is on a unit hypersphere . The temperature τ controls the concentration level of representations ; please refer to ( Wang & Liu , 2021 ) for detailed behaviors of τ in the contrastive loss . Intuitively , the InfoNCE loss aims to pull together the positive pair ( x , x+ ) from two different data augmentations of the same instance , and push x away from M negative examples of other instances . As discussed in ( Wang & Isola , 2020 ) , when M → ∞ , the InfoNCE loss in Eq . ( 1 ) can be approximately decoupled into two terms : alignment and uniformity , as shown in Eq . ( 2 ) . Despite the alignment term closes the positive pair , the key to avoiding representation collapse is the uniformity term , which makes the negative examples uniformly distributed on the hypersphere . Although beneficial , the negative examples inevitably lead to the class collision issue , hurting the representation learning for clustering . Non-contrastive learning . Non-contrastive learning-based methods have shown more promising results than contrastive learning for representation learning and downstream tasks ( Ericsson et al. , 2021 ) . Non-contrastive methods only optimize the alignment term in Eq . ( 2 ) to match the representations between two augmented views . Without negative examples , they leverage an online and a target network for two views , and use a predictor network to bridge the gap between these two views . They also stop the gradient of the target network to avoid the representation collapse . In particular , if τ = 0.5 , the loss used in ( Grill et al. , 2020 ; Chen & He , 2021 ) can be written as : Lnon−contr = −2g ( f ( x ) ) T f ′ ( x+ ) = ∥∥g ( f ( x ) ) − f ′ ( x+ ) ∥∥2 2 + const , ( 3 ) where g the predictor ; f and f ′ are the online and target networks , respectively ; the outputs of g ( f ( · ) ) and f ′ ( · ) are ` 2-normalized . However , as mentioned in ( Fetterman & Albrecht , 2020 ) , the non-contrastive learning methods often suffer from unstable training and highly rely on the batch-statistics and hyper-parameter tuning to avoid representation collapse . Even though Grill et al . ( 2020 ) ; Richemond et al . ( 2020 ) have proposed to use some tricks such as SyncBN ( Ioffe & Szegedy , 2015 ) and weight normalization ( Qiao et al. , 2019 ) to alleviate this issue , the additional computation cost is significant . Without negative examples , the collapse issue could be worsen when adding additional clustering losses for clustering task ; see Fig . A1 for the analysis of applying PCL ( Li et al. , 2021a ) to the BYOL . In a nutshell , most of existing successful deep clustering methods are based on contrastive learning for representation learning—giving rise to class collision issue—while the non-contrastive learning , due to unstable training with additional losses , is not yet ready for deep clustering . To that end , we explore the non-contrastive learning , i.e . BYOL , for deep clustering with positive sampling strategy and prototypical contrastive loss to avoid the class collision issue , improve the within-cluster compactness , and maximize the inter-class distance . 4 NON-CONTRASTIVE CLUSTERING Fig . 1 presents the overall framework of the proposed NCC . Based on BYOL , NCC is comprised of three networks : an online , a target , and a predictor . In Sec . 4.1 , we propose a positive sampling strategy to augment instance alignment to improve the within-cluster compactness . In Sec . 4.2 , a prototypical contrastive loss is introduced to maximize the inter-cluster distance using the pseudolabels from k-means clustering , which can encourage uniform representations . Finally , we formulate NCC into an EM framework to facilitate the understanding of training procedure in Sec . 4.3 .

Summary: This paper proposes a novel method of self-supervised representation learning. To circumvent the class collision issue arose from building a large set of negative samples in contrastive SSL based methods, it is built from non-contrastive SSL methods, such as BYOL. The goal is to handle the weaknesses of non-contrastive SSL methods, training instability and representation collapse. The two proposed methods for alleviating the two factors are i) augmented positive sampling and ii) optimizing uniformity of representation space via prototypical cluster features computed from k-means clustering.