You Never Cluster Alone

1 Introduction . Ancestored by various similarity-based [ 59 ] and feature-based [ 4 , 56 ] approaches , unsupervised deep clustering jointly optimizes data representations and cluster assignments [ 81 ] . A recent fashion in this domain takes inspiration from contrastive learning in computer vision [ 11 , 12 , 24 ] , leveraging the effectiveness and simplicity of discriminative feature learning . This strategy is experimentally reasonable , as previous research has found that the learnt representations reveal data semantics and locality [ 34 , 80 ] . Even a simple migration of contrastive learning significantly improves clustering performance , of which examples include a two-stage clustering pipeline [ 73 ] with contrastive pretraining and k-means [ 56 ] and a composition of an InfoNCE loss [ 62 ] and a clustering one [ 81 ] in [ 89 ] . Compared with the deep generative counterparts [ 16 , 36 , 54 , 86 ] , contrastive clustering is free from decoding and computationally practical , with guaranteed feature quality . However , have we been paying too much attention to the representation expressiveness of a single data point ? Intuitively , a standalone data point , regardless of its feature quality , can not tell us much about how the cluster looks like . Fig . 1 illustrates a simple analogy using the TwoMoons dataset . Without any context for the crescents , it is difficult to assign a data point to either of the two clusters based on its own representation , as the point can be inside one moon but still close to the other . Accordingly , observing more data reveals more about the holistic distributions of the clusters , e.g. , the shapes of the moons in Fig . 1 , and thus heuristically benefits clustering . Although we can implicitly parametrize the context of the clusters by the model itself , e.g. , using a Gaussian mixture model ( GMM ) [ 4 ] or encoding this information by deep model parameters , explicitly representing the context yields the most common deep learning practice . This further opens the door for learning cluster-level representations with all corresponding data points . Namely , you never cluster alone . ∗A part of this work was done when the author was with eBay . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , motivated by the thought experiment above , we develop a multi-granularity contrastive learning framework , which includes an instance-level granularity and a cluster-level one . The former learning track conveys the conventional functionalities of contrastive learning , i.e. , learning compact data representations and preserving underlying semantics . We further introduce a set of latent variables as cluster assignments , and derive an evidence lower bound ( ELBO ) of instance-level InfoNCE [ 62 ] . As per the cluster-level granularity , we leverage these latent variables as weights to aggregate all the corresponding data representations as the set representation [ 87 ] of a cluster . We can then apply contrastive losses to all clusters , thereafter rewarding/updating the cluster assignments . Abbreviated as twin-contrast clustering ( TCC ) , our work delivers the following contributions : • We develop the novel TCC model , which , for the first time , shapes and leverages a unified representation of the cluster semantics in the context of contrastive clustering . • We define and implement the cluster-level augmentations in a batch-based training and stochastic labelling procedure , which enables on-the-fly contrastive learning on clusters . • We achieve significant performance gains against the state-of-the-art methods on five benchmark datasets . Moreover , TCC can be trained from scratch , requiring no pre-trained models or auxiliary knowledge from other domains . 2 Preliminaries . 2.1 Contrastive Learning . Contrastive learning , as the name suggests , aims to distinguish an instance from all others using embeddings , with the dot-product similarity typically being used as the measurement . Let X = { xi } Ni=1 be an N -point dataset , with the i-th observation xi ∈ Rdx . An arbitrary transformation f : Rdx → Rdm encodes each data point into a dm-dimensional vector . With the index i being the identifier , the InfoNCE loss [ 62 ] discriminates xi from the others through a softmax-like likelihood : log p ( i|xi ) = log exp ( vᵀi f ( xi ) /τ ) ∑N j=1 exp ( vᵀj f ( xi ) /τ ) . ( 1 ) A temperature hyperparameter τ controls the concentration level [ 26 , 80 ] . V = { vi } Ni=1 refers to the vocabulary of the dataset , which is usually based on the data embeddings under different augmentations . As caching the entire V is not practical for large-scale training , existing works propose surrogates of Eq . ( 1 ) , e.g. , replacing V with a memory bank [ 80 ] , queuing it with a momentum network [ 12 , 24 ] , or training with large batches [ 11 ] . 2.2 Deep Set Representations . To learn the representations of sets , we need to consider permutation-invariant transformations . Zaheer et al . [ 87 ] showed that all permutation-invariant functions T ( · ) applied to a set X generally fall into the following form : T ( X ) = g ( ∑N i=1 h ( xi ) ) , ( 2 ) where g ( · ) and h ( · ) are arbitrary continuous transformations . Note that the aggregation above can be executed on a weighted basis , which is typically achieved by the attention mechanism between the set-level queries and instance-level keys/values [ 31 , 49 ] . Our design on each cluster is partially inspired by this , as back-propagation does not support hard instance-level assignment . Next , we define our cluster representation along with the clustering procedure , and describe how it is trained with contrastive learning . 3 Twin-Contrast Clustering . We consider a K-way clustering problem , with K being the number of clusters . Let k ∈ { 1 , · · · , K } denote the entry of a cluster that xi may belong to , and the categorical variable πi = [ πi ( 1 ) , · · · , πi ( K ) ] indicate the cluster assignment probabilities of xi . Following common practice , we regard image clustering as our target for simplicity . Fig . 2 provides a schematic of TCC . The cluster-level contrast track reflects our motivation from Sec . 1 , while the instance-level one learns the semantics of each image . We bridge these two tracks with the inference model πi ( k ) = qθ ( k|xi ) so that both losses reward and update the assignment on each xi . In particular , qθ ( k|xi ) is parametrized by a softmax operation : πi ( k ) = qθ ( k|xi ) = exp ( µᵀk fθ ( xi ) ) ∑K k′=1 exp ( µ ᵀ k′ fθ ( xi ) ) , ( 3 ) where fθ ( · ) is a convolutional neural network ( CNN ) [ 47 , 48 ] built upon random data augmentations , producing dm-dimensional features . We denote µθ = { µk } Kk=1 as a set of trainable cluster prototypes , where θ refers to the collection of all parameters . In Sec . 3.1 , we leverage qθ ( k|xi ) to aggregate cluster features , and in Sec . 3.2 we derive the ELBO of Eq . ( 1 ) with qθ ( k|xi ) . In the following , we will omit the index i for brevity when x and π clearly correspond to a single data point . 3.1 Representing and Augmenting the Context for Cluster-Level Contrast . Cluster-Level Representation We implement Eq . ( 2 ) for each cluster using soft aggregation , where qθ ( k|x ) weighs the relevance of the data to the given cluster . Denoted as rk , the k-th cluster representation is computed by : hθ ( x ; k ) = π ( k ) · fθ ( x ) , rk = Tθ ( X ; k ) = ( ∑N i=1 hθ ( xi ; k ) ) /∥∥∥∥∑Ni=1 hθ ( xi ; k ) ∥∥∥∥ 2 , ( 4 ) where ‖·‖2 refers to the L2-norm . We adopt L2 normalization here for two main purposes . First , the summation of π ( k ) = qθ ( k|x ) along x is not self-normalized . Second , and more importantly , as shown in [ 11 , 12 , 24 ] , L2-normalized features benefit contrastive learning . The Anchored Cluster Semantics Intuitively , π ( k ) reflects the degree of relevance of a datum to the k-th cluster . With it being the aggregation weight , rk represents the information that is related to the corresponding prototype µk . In other words , our design treats each µk as the semantic anchor that queries the in-coming batch to form a representation describing a certain latent topic . Cluster-Level Augmentation Equivalents Contrastive learning is usually employed alongside random data augmentation [ 11 , 24 ] to obtain positive candidates . Though defining a uniform augmentation scheme for sets is beyond the scope of this paper , the proposed model reflects clusterlevel augmentation in its design by the following heuristics : ( a ) Augmentation on elements . TCC implicitly inherits existing image augmentation techniques ( such as cropping , color jittering , random flipping , and grayscale conversion ) by implementing them using fθ ( · ) . ( b ) Irrelevant minorities . We consider injecting a small proportion of irrelevant data into a cluster representation , while keeping the main semantics of the cluster unchanged . Eq . ( 4 ) turns out to be an equivalent to this . As the softmax product π ( k ) is always positive , those data that are not very related to the given cluster still contribute to the cluster ’ s representation , which compiles the irrelevant . Meanwhile , these irrelevant data are not dominating the value of the cluster representation , because the small value of π ( k ) scales the feature magnitude during aggregation , which counts the minority . ( c ) Subsetting . Empirically , a subset of a cluster holds the same semantics as the original . Batch-based training samples data at each step , which naturally creates subsets for each cluster . We experimentally find the above augmentation equivalents are sufficient for the clustering task . On the other hand , since Eq . ( 4 ) is permutation-invariant , reordering the sequence of data does not yield a valid augmentation . Cluster-Level Contrastive Learning We define a simple contrastive objective that preserves the identity of each cluster against the rest . Having everything in a batch , e.g. , a SimCLR-like framework [ 11 ] , does not allow augmentation ( c ) to be fully utilized in the loss , since the two augmented counterparts rk and r̂k from a batch may form part of the same subset of a cluster . Hence , we opt for the MoCo-like solution [ 24 ] , employing an L-sized memory queue P = { pl } Ll=1 to cache negative samples and a momentum network to produce r̂k = Tθ̂ ( X ; k ) . P stores each cluster representation under different subsets , training with which preserves the temporal semantic consistency [ 45 ] of clusters . Our cluster-level objective minimizes the following negative loglikelihood ( NLL ) : L1 = Ek [ − log pθ ( k|rk ) ] = − 1 K K∑ k=1 log exp ( r̂ᵀkrk/τ ) exp ( r̂ᵀkrk/τ ) + ∑L l=1 exp ( p ᵀ l rk/τ ) 1 ( l mod K 6= k ) , ( 5 ) where 1 ( · ) is an indicator function and mod is the modular operator . Since the cluster number K can be less than the queue size L , we exclude the features that represent the same cluster as k in the negative sample collection P by inserting the indicator function into the loss above .

Summary: The authors propose a method for end-to-end deep clustering. Authors utilizes NCE loss and proposes a method to produce cluster level loss. They motivate their approach and supply empirical evaluation.