Multi-view Contrastive Graph Clustering

1 Introduction . An attributed graph contains of node features and edges characterizing the pairwise relations between nodes . It is a natural and effcient representation for many real-world data [ Liu et al. , 2021 ] . For example , social network users have their own profles and the topological graph refects their social relationships . Different from most classical clustering methods like K-means and hierarchical clustering which only focus on Euclidean data , graph clustering divides unlabeled nodes of graph into clusters . Typical graph clustering methods frst learn a good representation of graph and then apply a classical clustering method upon the embeddings . For example , large-scale information network embedding ( LINE ) [ Tang et al. , 2015 ] is a popular graph representation learning method , which can preserve both local and global information and scale up easily to large-scale networks . To incorporate node features and graph structure information , graph autoencoder ( GAE ) [ Kipf and Welling , 2016 ] employs a graph convolution network ( GCN ) encoder and achieves signifcant performance improvement . The real-life data are often collected from various sources or obtained from different extractors , thus are naturally represented by different features or views [ Kang et al. , 2021 , 2020c ] . Each view could be noisy and incomplete , but important factors , such as geometry and semantics , tend to be shared among all views . Therefore , features and graphs of different views are complementary , which implies that it ’ s paramount to integrate all features and graphs of diverse views to improve the performance of clustering task . Numerous graph-based multi-view clustering methods have been developed to capture the consensus information shared by different views in the literature . Graph-based multi-view clustering constructs a graph for each view and fuses them based on a weighting mechanism [ Wang et al. , 2019 ] . Multi-view spectral clustering network [ Huang et al. , 2019 ] learns a discriminative representation by using a deep metric learning network . These methods are developed for feature matrix and can not handle graph data . To directly process graph data , some representative methods have also been proposed . ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Scalable multiplex network embedding ( MNE ) [ Zhang et al. , 2018 ] is a scalable multi-view network embedding model , which learns multiple relations by a unifed network embedding framework . Principled multilayer network embedding ( PMNE ) [ Liu et al. , 2017 ] proposes three strategies ( “ network aggregation ” , “ results aggregation ” , and “ layer co-analysis ” ) to project a multilayer network into a continuous vector space . Nevertheless , they fail to explore the feature information [ Lin and Kang , 2021 ] . Recently , based on GCN , One2Multi graph autoencoder clustering ( O2MA ) framework [ Fan et al. , 2020 ] and multi-view attribute GCNs for clustering ( MAGCN ) [ Cheng et al. , 2020 ] achieve superior performance on graph clustering . O2MA introduces a graph autoencoder to learn node embeddings based on one informative graph and reconstruct multiple graphs . However , the shared feature representation of multiple graphs could be incomplete because O2MA only takes into account the informative view selected by modularity . MAGCN exploits the abundant information of all views and adopts a cross-view consensus learning by enforcing the representations of different views to be as similar as possible . Nevertheless , O2MA targets for multiple graphs while MAGCN mainly solves graph structured data of multi-view attributes . They are not directly applicable to multiple graphs data with multi-view attributes . Therefore , the research of multi-view graph clustering is at the initial stage and more dedicated efforts are pressingly needed . In this paper , we propose a generic framework of clustering on attributed graph data with multiview features and multiple topological graphs , denoted by Multi-view Contrastive Graph Clustering ( MCGC ) . To be exact , MCGC learns a new consensus graph by exploring the holistic information among various attributes and graphs rather than utilizing the initial graph . The reason of introducing graph learning is that the initial graph is often noisy or incomplete , which leads to suboptimal solutions [ Chen et al. , 2020b , Kang et al. , 2020b ] . A contrastive loss is adopted as regularization to make the consensus graph clustering-friendly . Moreover , we implement on the smooth representation rather than raw data . The contributions of this work could be summarized as follows : • To boost the quality of learned graph , we propose a novel contrastive loss at graph-level . It is capable of drawing similar nodes close and pushing those dissimilar ones apart . • We propose a generic clustering framework to handle multilayer graphs with multi-view attributes , which contains graph fltering , graph learning , and graph contrastive components . The graph fltering is simple and effcient to obtain a smoothed representation ; the graph learning is utilized to generate the consensus graph with an adaptive weighting mechanism for different views . • Our method achieves state-of-the-art performance compared with shallow methods and deep methods on fve benchmark datasets . 2 Related Work . 2.1 Multi-view Clustering . Large quantities of multi-view clustering methods have been proposed in the last decades . Multi-view low-rank sparse subspace clustering [ Brbić and Kopriva , 2018 ] obtains a joint subspace representation across all views by learning an affnity matrix constrained by sparsity and low-rank constraint . Cross-view matching clustering ( COMIC ) [ Peng et al. , 2019 ] enforces the graphs to be as similar as possible instead of the representation in the latent space . Robust multi-view spectral clustering ( RMSC ) [ Xia et al. , 2014 ] uses a shared low-rank transition probability matrix derived from each single view as input to the standard Markov chain method for clustering . These methods are designed for feature matrix and try to learn a graph from data . To directly cluster multiple graphs , self-weighted multi-view clustering ( SwMC ) [ Nie et al. , 2017 ] method learns a shared graph from multiple graphs by using a novel weighting strategy . Above methods are suitable for graph or feature data only , and can not simultaneously explore attributes and graph structure . As previously discussed , O2MA [ Fan et al. , 2020 ] and MAGCN [ Cheng et al. , 2020 ] can handle attributed graph , but they are not direct applicable to generic multi-view graph data . 2.2 Contrastive Clustering . Due to its impressive performance in many tasks , contrastive learning has become the most hot topic in unsupervised learning . Its motivation is to maximize the similarity of positive pairs and H > − H > S 2 2min + α kSkF , FS distance of negative pairs [ Hadsell et al. , 2006 ] . Generally , the positive pair are composed of data augmentations of the same instance while those of different instances are regarded as negatives . Several loss functions have been proposed , such as the triplet loss [ Chopra et al. , 2005 ] , the noise contrastive estimation ( NCE ) loss [ Gutmann and Hyvärinen , 2010 ] , the normalized temperature-scaled cross entropy loss ( NT-Xent ) [ Chen et al. , 2020a ] . Deep robust clustering turns maximizing mutual information into minimizing contrastive loss and achieves signifcant improvement after applying contrastive learning to decrease intra-class variance [ Zhong et al. , 2020 ] . Contrastive clustering develops a dual contrastive learning framework , which conducts contrastive learning at instance-level as well as cluster-level [ Li et al. , 2021 ] . As a result , it produces a representation that facilitates the downstream clustering task . Unfortunately , theses method can only handle single-view data . Recently , by combining reconstruction , cross-view contrastive learning , and cross-view dual prediction , incomplete multi-view clustering via contrastive prediction ( COMPLETER ) [ Lin et al. , 2021a ] performs data recovery and consistency learning of incomplete multi-view data simultaneously . It also obtains promising performance on complete multi-view data . However , it can not deal with graph data . On the other hand , contrastive multi-view representation learning on graphs ( MVGRL ) [ Hassani and Khasahmadi , 2020 ] method is proposed , which performs representation learning by contrasting two diffusion matrices transformed from the adjacency matrix . It reports better performance than variational GAE ( VGAE ) [ Kipf and Welling , 2016 ] , marginalized GAE ( MGAE ) [ Wang et al. , 2017 ] , adversarially regularized GAE ( ARGA ) and VGAE ( ARVGA ) [ Pan et al. , 2018 ] , and GALA [ Park et al. , 2019 ] . Different from MVGRL , our contrastive regularizer is directly applied on learned graph . 3.1 Notation . Defne the multi-view graph data as G = { V , E1 , ... , EV , X1 , ... , XV } , where V represents the sets of N nodes , eij ∈ Ev denotes the relationship between node i and node j in the v-th view , v vXv = { x1 , ... , x } > is the feature matrix . Adjacency matrices { Aev } V characterize the initial N v=1 graph structure . { Dv } V represent the degree matrices in various views . The normalized adjacency 3 Methodology 1 v=1 matrix Av = ( Dv ) − 2 ( Aev + I ) ( Dv ) − 1 2 and the corresponding graph laplacian Lv = I − Av . 3.2 Graph Filtering . A feature matrix X ∈ RN×d of N nodes can be treated as d N -dimensional graph signals . A natural signal should be smooth on nearby nodes in term of the underlying graph . The smoothed signals H can be achieved by solving the following optimization problem [ Zhu et al. , 2021 , Lin et al. , 2021b ] � min kH − Xk2 F + s Tr H > LH , H ( 1 ) where s > 0 is a balance parameter and L is the laplacian matrix associated with X . H can be obtained by taking the derivative of Eq . ( 1 ) w.r.t . H and setting it to zero , which yields H = ( I + sL ) −1X . ( 2 ) To get rid of matrix inversion , we approximate H by its frst-order Taylor series expansion , i.e. , H = ( I − sL ) X . Generally , m-th order graph fltering can be written as H = ( I − sL ) mX , ( 3 ) where m is a non-negative integer . Graph fltering can flter out undesirable high-frequency noise while preserving the graph geometric features . 3.3 Graph Learning . Since real-world graph is often noisy or incomplete , which will degrade the downstream task performance if it is directly applied . Thus we learn an optimized graph S from the smoothed representation H . This can be realized based on the self-expression property of data , i.e. , each data point can be represented by a linear combination of other data samples [ Lv et al. , 2021 , Ma et al. , 2020 ] . And the combination coeffcients represent the relationships among data points . The objective function on single-view data can be mathematically formulated as ( 4 ) ⎪ P⎪ V n exp ( t−1 ) S⎨ λv −1 + ij , j v , K ( t− 1 ) if in Ni ( t ) r = v=1 2 ( t−1 ) ⎪ V P ⎩⎪ n exp S λv ij , otherwise . K ( t−1 ) v=1 V NX X X2 exp ( Sij ) min λv ⎝ Hv > − Hv > S + α − log ⎠ . F PNS exp ( Sip ) v=1 i=1 j∈Nv p= ii ⎛ ⎞ V N VX X X X2 exp ( Sij ) γmin λv ⎝ Hv > − Hv > S + α − log ⎠ + ( λv ) . F PNS , λv exp ( Sip ) v=1 i=1 j∈Nv p= i v=1i NX X exp ( Sij ) J = − log PN , exp ( Sip ) i=1 j∈Nv p= ii V VX X2 γmin λv Hv > − Hv > S + αkSk2 + ( λv ) , FFS , λv v=1 v=1 where S ∈ RN×N is the graph matrix and α > 0 is the trade-off parameter . The frst term is the reconstruction loss and the second term serves as a regularizer to avoid trivial solution . Many other regularizers could also be applied , such as the nuclear norm , sparse ` 1 norm [ Kang et al. , 2020a ] . To tackle multi-view data , we can compute a smooth representation Hv for each view and extend Eq . ( 4 ) by introducing a weighting factor to distinguish the contributions of different views ( 5 ) where λv is the weight of v-th view and γ is a smooth parameter . Eq . ( 5 ) learns a consensus graph S shared by all views . To learn a more discriminative S , we introduce a novel regularizer in this work .

Summary: This paper proposes a multi-view clustering method. They first use graph filter to generate a new node representation for input, and then do self-expression in a linear combination of various views. Afterwards, they propose a graph contrastive regularizer, which mimics the contrastive learning given the nearest neighbors as positive samples and any others as negative ones. Experiments compared with recent works and ablation studies are presented.