Learning Representations by Contrasting Clusters While Bootstrapping Instances

1 INTRODUCTION . Learning to extract generalized representations from a high-dimensional image is essential in solving various down-stream tasks in computer vision . Though a supervised learning framework has shown to be useful in learning discriminative representations for pre-training the model , expensive labeling cost makes it practically infeasible in a large-scale dataset . Moreover , relying on the human-annotated labels tends to cause several issues such as class imbalance ( Cui et al. , 2019 ) , noisy labels ( Lee et al. , 2019 ) , and biased datasets ( Bahng et al. , 2019 ) . To address these issues , self-supervised visual representation learning , which does not require any given labels , has emerged as an alternative training framework , being actively studied to find a proper training objective . Recently , self-supervised approaches with contrastive learning ( Wu et al. , 2018 ; Chen et al. , 2020a ; He et al. , 2020 ) have rapidly narrowed the performance gap with supervised pre-training in various vision tasks . The contrastive method aims to learn invariant mapping ( Hadsell et al. , 2006 ) and instance discrimination . Intuitively , two augmented views of the same instance are mapped to the same latent space while different instances are pushed away . However , aforementioned instance discrimination does not consider the semantic similarities of the representations ( e.g. , same class ) , even pushing away the relevant instances . This affects the learned representations to exhibit uniformly distributed characteristics , proven by the previous works ( Wang & Isola , 2020 ; Chen & Li , 2020 ) . We point out that this uniformly distributed characteristic over instances can be a fundamental limitation against improving the learned representation quality . For instance , consider the representations illustrated in Fig . 1 . It indicates a simple case where linearly separable representations do not always guarantee that they can be properly clustered , which is not appropriate for non-discriminative downstream tasks such as information retrieval , density estimation , and cluster analysis ( Wu et al. , 2013 ) . In response , we start this work by asking : How can we learn the representations to be properly clustered even without the class labels ? In this work , we propose a self-supervised training framework that makes the learned representations not only linearly separable but also properly clustered , as illustrated in Fig . 2 . To mitigate the uniformly distributed constraint while preserving the invariant mapping , we replace the instance discrimination with an instance alignment problem , pulling the augmented views from the same instance without pushing away the views from the different images . However , learning the invariant mapping without discrimination can easily fall into a trivial solution that maps all the individual instances to a single point . To alleviate this shortcoming , we adopt a bootstrapping strategy from Grill et al . ( 2020 ) , utilizing the Siamese network , and a momentum update strategy ( He et al. , 2020 ) . In parallel , to properly cluster the semantically related instances , we are motivated to design additional cluster branch . This branch aims to group the relevant representations by softly assigning the instances to each cluster . Since each of cluster assignments needs to be discriminative , we employ the contrastive loss to the assigned probability distribution over the clusters with a simple entropy-based regularization . In the meantime , we constructed the cluster branch in multi-scale clustering starategy where each head deals with a different number of clusters ( Lin et al. , 2017 ) . Since there exists a various granularity of semantic information in images , it helps the model to effectively capture the diverse level of semantics as analyzed in Section 4.5 . In summary , our contributions are threefold , as follows : • We propose a novel self-supervised framework which contrasts the clusters while bootstrapping the instances that can attain both linearly separable and clusterable representations . • We present a novel cluster branch with multi-scale strategy which effectively captures the different levels of semantics in images . • Our method empirically achieves state-of-the-art results in CIFAR-10 , CIFAR-100 , and STL-10 on representation learning benchmarks , for both classification and clustering tasks . 2 RELATED WORK . Our work is closely related to unsupervised visual representation learning and unsupervised image clustering literature . Although both have a slightly different viewpoints of the problem , they are essentially similar in terms of its goal to find good representations in unlabelled datasets . Instance-level discrimination utilizes an image index as supervision because it is an unique signal in the unsupervised environment . NPID ( Wu et al. , 2018 ) firstly attempts to convert the classwise classification into the extreme of instance-wise discrimination by using external memory banks . MoCo ( He et al. , 2020 ) replaces the memory bank by introducing a momentum encoder that memorizes knowledge learned from the previous mini-batch . SimCLR ( Chen et al. , 2020a ) presents that it is crucial for representation quality to combine data augmentations using a pretext head after the encoder . Although recent studies show promising results on benchmark datasets , the instancewise contrastive learning approach has a critical limitation that it pushes away representations from different images even if the images have similar semantics , e.g. , belonging to the same class . Cluster-level bootstrapping is an alternative paradigm that enhancing the initial bias of the networks can be useful in obtaining a discriminative power in visual representations , since convolutional neural networks work well on capturing the local patterns ( Caron et al. , 2018 ) . In the case of using pseudo-labels , K-means ( Caron et al. , 2018 ) or optimal transport ( Asano et al. , 2019 ; Caron et al. , 2020 ) are commonly adopted for clustering . On the other hand , soft clustering methods have also been actively studied to allow flexible cluster boundaries ( Ji et al. , 2019 ; Huang et al. , 2020 ) . Recently , a 2-stage training paradigm has been proposed to construct the cluster structure initialized from the representations learned by instance discrimination ( Gansbeke et al. , 2020 ) . 3 METHOD . Our work is motivated by an observation from SupCLR ( Khosla et al. , 2020 ) , which additionally pulls the representations together from different instances by using groundtruth labels . However , directly applying this idea in an unsupervised environment with pseudo-labels is challenging , because small false-positive errors at the initial step can be gradually spread out , degrading the quality of final representations . Instead , the main idea of our approach avoid pushing away those instances close enough to each other . To validate this idea , we conducted a toy experiment that a pulling force is only corresponding to two augmented views of the same image while not pushing the images within the same class by using the groundtruth label . We found that its classification accuracy increases over 5 % on STL-10 datasets compared to that of SimCLR ( Chen et al. , 2020a ) . Inspired by this experiment , we design our model ( i ) not to push away relevant instances with our instance-alignment loss ( Section 3.2 ) while ( ii ) discriminating the representations in a cluster-wise manner . ( Section 3.3-3.4 ) . 3.1 PRELIMINARIES . As shown in Fig . 3 , we adopt stochastic data augmentation algorithms ( Chen et al. , 2020a ; He et al. , 2020 ; Chen et al. , 2020b ; Caron et al. , 2020 ) to generate two different augmented views x′i and x′′i of the same image xi ∼ X = { x1 , x2 , ... , xN } where N is the number of unlabelled images . Inspired by Luo et al . ( 2018 ) ; Grill et al . ( 2020 ) , C2BIN consists of an instance predictor P a ( · ) , cluster predictors P c , k ( · ) , and two Siamese networks called the runner Eθ ( · ) and the follower Eφ ( · ) , respectively . The runner Eθ is rapidly updated to find the optimal parameters θ∗ over the search spaces , while the follower Eφ generates the target representations for the Eθ . The Eθ is composed of the two neural functions : encoder Fθ ( · ) and instance projector Gaθ ( · ) , and vice versa for the follower Eφ . To bootstrap the instance-level alignment , Eθ , Eφ , and P a are used . Afterwards , Fθ and P c , k are utilized to contrast the cluster-wise features . 3.2 BOOTSTRAPPING LOSS OF INSTANCE REPRESENTATIONS . Given an image x ∼ X , we can obtain two augmented views x′ = t′ ( x ) and x′′ = t′′ ( x ) where t′ and t′′ are sampled from a set of stochastic data augmentations T as mentioned above . Even though augmented views are distorted , they should contain similar semantics , and the learned representations should be closely aligned in the latent space . For training , we forward x′′ through the follower Eφ to obtain target representations at an instance level ; the runner Eθ aims to make the embedding vector of x′ closer to them . That is , we first extract image representations r = Fθ ( x′ ) ∈ Rdr where dr is the number of dimensions of our representations . Afterwards , we introduce a pretext-specific instance-wise projector Gaθ ( · ) and then obtain pretext embedding vectors za = Gaθ ( r ) ∈ R1×da ; the target pretext vectors ẑa can be obtained using the same procedure by Eφ . Motivated from Grill et al . ( 2020 ) , we calculate our alignment loss as the cosine distance as Lalign = 1− P a ( za ) · ẑa ||P a ( za ) ||2||ẑa||2 , ( 1 ) where P a ( za ) , ẑa ∈ R1×da and we adopt the number of dimensions of projected features da as in Chen et al . ( 2020a ; c ) . 3.3 CONTRASTIVE LOSS OF BATCH-WISE CLUSTER ASSIGNMENTS . Our high-level motivation of this branch is that an image feature r can be represented as the combination of cluster features capturing local patterns . However , grouping similar images conflict with the instance-level invariant mapping ; therefore , we introduce an additional branch which contains cluster predictor P c , k ( · ) after the encoder Fθ ( · ) . The cluster predictor P c , k is a linear function whose takes ri as an input and transform it to a K-dimensional output vector . Therefore , zci = P c , k ( ri ) represents a degree of confidence for the i-th image representations ri to belong to the k-th cluster feature , i.e. , zci = [ z c i,1 , z c i,2 , ... , z c i , k ] ∈ R1×K , ( 2 ) where zci indicate a cluster membership distribution of the given image xi . Since we sample n items for training , Zc ∈ Rn×K is the set of memberships distribution of the given mini-batch . Now we define batch-wise cluster assignment vectors ( BCAs ) ck as ck = Z c : ,k = z c 1 , k ... zcn , k ∈ Rn×1 , ( 3 ) which indicates how much the k-th cluster is mapped by images in the mini-batch . Although ck will dynamically change as a new mini-batch is given , the same cluster features between differently augmented views from the same image should be similar while pushing away the others to capture diverse patterns . To this end , we simply utilize the contrastive loss between the BCAs as Lbcaclust = 1 K K∑ i=1 − log ( exp ( c′i · c′′i /τ ) ∑K j=1 1 [ j 6=i ] exp ( c ′ i · c′′j /τ ) ) , ( 4 ) where τ indicates a temperature value . The vectors c′ and c′′ are outputs of P c , k following the encoder Fθ by taking x′ and x′′ respectively . Unfortunately , most of the clustering-based methods suffers from falling into a degenerate solution where the majority of items are allocated in a few clusters , especially in an unsupervised environment . To mitigate this issue , we first compute the mass of assignment to k-th cluster as sk = ∑N i ck ( i ) where ck ( i ) , indicating each element of ck . Afterwards , we encourage ri to be stochastically activated for diverse cluster features as much as possible by maximizing an entropy of s. To this end , we formulate the cluster loss function as Lclust = Lbcaclust − λentH ( s ) , ( 5 ) where H indicates an entropy function as H ( s ) = − ∑K i si log si and λent is the weight value for the regularization term .

Summary: In this paper, the authors augment the instance-level self-supervised learning with cluster-aware learning mechanism during the training procedure. Specifically, for each training batch, the authors project the instances into a clustering space and then utilize a cluster-aware contrastive loss to push the augmented samples from the same instance to belong to the same cluster, otherwise for different instances. To ensure the clustering not to collapse into a single or a few cluster to find the trivial solutions, the authors further add a penalization item keep the entropy of clustering assignment be uniform to some extent. The experimental results demonstrate that the proposed method can improve the representation learning performance over SoTA methods on several datasets, while also outperforms the previous methods on clustering task. Further ablation studies show that the loss is effective to ensure the learned representation more discriminative and clusterable.