Multi-view Contrastive Graph Clustering
Authors: Erlin Pan
zhao kang
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-view clustering
contrastive learning
graph learning
subspace clustering

Abstract: With the explosive growth of information technology, multi-view graph data have become increasingly prevalent and valuable. Most existing multi-view clustering techniques either focus on the scenario of multiple graphs or multi-view attributes. In this paper, we propose a generic framework to cluster multi-view attributed graph data. Specifically, inspired by the success of contrastive learning, we propose multi-view contrastive graph clustering (MCGC) method to learn a consensus graph since the original graph could be noisy or incomplete and is not directly applicable. Our method composes of two key steps: we first filter out the undesirable high-frequency noise while preserving the graph geometric features via graph filtering and obtain a smooth representation of nodes; we then learn a consensus graph regularized by graph contrastive loss. Results on several benchmark datasets show the superiority of our method with respect to state-of-the-art approaches. In particular, our simple approach outperforms existing deep learning-based methods.

Multi-view Contrastive Graph Clustering
Erlin Pan, Zhao Kang"
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China mujisixsix6dgmal. com zkang@uestc. edu.
Abstract
With the explosive growth of information technology, multi-view graph data have become increasingly prevalent and valuable. Most existing multi-view clustering techniques either focus on the scenario of multiple graphs or multi-view attributes. In this paper, we propose generic framework to cluster multi-view attributed graph data. Specifically, inspired by the success of contrastive learning, we propose multi-view contrastive graph clustering (MCGC) method to learn a consensus graph since the original graph could be noisy or incomplete and not directly applicable. Our method composes of two key steps: first filter out the undesirable high- frequency noise while preserving the graph geometric features via graph filtering and obtain smooth representation of nodes; then learn consensus graph regularized by graph contrastive loss. Results on several benchmark datasets show the superiority of our method with respect to state-of-the-ee approaches. In particular, our simple approach outperforms existing deep learning- -based methods.
Introduction
An attributed graph contains of node features and edges characterizing the pairwise relations between nodes. It is a natural and efficient representation for many real-world data [Liu et al., [02T]. For example, social network users have their own profiles and the topological graph reflects their social relationships. Different from most classical clustering methods like K-means and hierarchical clustering which only focus on Fuclidean data, graph clustering divides unlabeled nodes of graph into clusters. Typical graph clustering methods first learn a good representation of graph and then apply classical clustering method upon the embeddings. For example, large-scale information network embedding (LINE) [Tang et al. 2015) is popular graph representation learning method, which can preserve both local and global information and scale up easily to large-scale networks. To incorporate node features and graph structure information, graph autoencoder (GAE) (Kipf and Welling 2016 employs graph convolution network (GCN) encoder and achieves significant performance improvement. The real-life data are often collected from various sources or obtained from different extractors, thus naturally represented by different features or views (Kang et al.) 2021 2020c). Each view could be noisy and incomplete, but important factors, such as geometry and semantics, tend to be shared among all views. Therefore, features and graphs of different views are complementary. which implies that it's paramount to integrate all features and graphs of diverse views to improve the performance of clustering task.
Numerous graph-based multi-view clustering methods have been developed to capture the consensus information shared by different views the literature. Graph-based multi-view clustering constructs graph for each view and fuses them based on weighting mechanism [Wang et al. 2019]. Multi-view spectral clustering network (Huang et al. 2019] learns a discriminative representation by using deep metric learning network. These methods are developed for feature matrix and can not handle graph data. To directly process graph data, some representative methods have also been proposed.
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Scalable multiplex network embedding (MNE) [Zhang et al., 2018) is a scalable multi-view network embedding model, which learns multiple relations by a unified network embedding framework. Principled multilayer network embedding (PMNE) (Liu et al. [017] proposes three strategies ("network aggregation". "results aggregation", and "layer co-analysis") to project a multilayer network into a continuous vector space. Nevertheless, they fail to explore the feature information
[Lin and Kang 2021].
Recently, based on GCN, One2Multi graph autoencoder clustering (02MA) framework [aan et al.
2020] and multi-view attribute GCNs for clustering (MAGCN) [Cheng et al. 2020) achieve superior performance on graph clustering. O2MA introduces a graph autoencoder to learn node embeddings based on one informative graph and reconstruct multiple graphs. However, the shared feature representation of multiple graphs could be incomplete because O2MA only takes into account the informative view selected by modularity. MAGCN exploits the abundant information of all views and adopts cross-view consensus learning by enforcing the representations of different views to be as similar as possible. Nevertheless, O2MA targets for multiple graphs while MAGCN mainly solves graph structured data of multi-view attributes. They not directly applicable to multiple graphs data with multi-view attributes. Therefore, the research of multi-view graph clustering is at the initial stage and more dedicated efforts are pressingly needed.
In this paper, we propose a generic framework of clustering on attributed graph data with multi- view features and multiple topological graphs, denoted by Multi-view Contrastive Graph Clustering (MCGC). To be exact, MCGC learns a new consensus graph by exploring the holistic information among various attributes and graphs rather than utilizing the initial graph. The reason of introducing graph learning is that the initial graph is often noisy or incomplete, which leads to suboptimal solutions [Chen et al., 2020b Kang et al., 2020b]. A contrastive loss is adopted as regularization to make the consensus graph clustering- friendllyridll. Moreover, we implement on the smooth representation rather than raw data. The contributions of this work could be summarized as follows:
To boost the quality of learned graph, we propose novel contrastive loss at graph-level. It is capable of drawing similar nodes close and pushing those dissimilar ones apart.
We propose a generic clustering framework to handle multilayer graphs with multi-view attributes, which contains graph filtering, graph learning, and graph contrastive components. The graph filtering is simple and efficient to obtain a smoothed representation; the graph learning utilized to generate the consensus graph with an adaptive weighting mechanism for different views.
Our method achieves state-of-ther art performance compared with shallow methods and deep methods five benchmark datasets.
2 Related Work
2.1 Multi-view Clustering
Large quantities of multi-view clustering methods have been proposed in the last decades. Multi-view low-rank sparse subspace clustering (Brbié and Kopriva 2018 obtains a joint subspace representation across all views by learning an affinity matrix constrained by sparsity and low-rank constraint. Cross-viee matching clustering (COMIC) (Peng al. 2019) enforces the graphs to be as similar as possible instead of the representation in the latent space. Robust multi-view spectral clustering (RMSC) Xia et al., 2014] uses a shared low-rank transition probability matrix derived from each single view as input to the standard Markov chain method for clustering. These methods are designed for feature matrix and try to learn graph from data. To directly cluster multiple graphs, self-weighted multi-view clustering (SwMC) [Nic et al.) 2017] method learns shared graph from multiple graphs by using a novel weighting strategy. Above methods are suitable for graph or feature data only, and can not simultancously explore attributes and graph structure. As previously discussed, O2MA [Fan al. 2020) and MAGCN [Cheng et al. 2020 can handle attributed graph, but they are not direct applicable to generic multi-view graph data.
2.2 Contrastive Clustering
Due to its impressive performance in many tasks, contrastive learning has become the most hot topic in unsupervised learning. Its motivation is to maximize the similarity of positive pairs and

---

distance of negative pairs [Hadsell et al. 2006]. Generally, the positive pair are composed of data augmentations of the same instance while those of different instances are regarded negatives. Several loss functions have been proposed, such as the triplet loss [Chopra et al. 2005). the noise con- trastive estimation (NCE) loss [Gutmann and Hyvärinen 2010]. the normalized temperature-scall cross entropy loss (NT--eent) [Chen et al., 2020a]] Deep robust clustering turns maximizing mutual information into minimizing contrastive loss and achieves sigmificant improvement after applying contrastive learning to decrease intra-class variance (Zhong et al. 20220]. Contrastive clustering develops a dual contrastive learning framework, which conducts contrastive learning at instance-level well as cluster-level [Li et al. 2021] J. As result, it produces representation that facilitates the downstream clustering task. Unfortunately, theses method can only handle single- view data.
Recently, by combining reconstruction, cross- sross-view contrastive learning, and cross-vir cross-view dual predic- tion, incomplete multi-view clustering via contrastive prediction (COMPLETER) [Lin et al. [22 performs data recovery and consistency learning of incomplete multi-view data simultaneously. It also obtains promising performance on complete multi-view data. However, can not deal with graph data. On the other hand, contrastive multi-view representation learning on graphs (MVGRL) [Hassani and Khasahmadi] 2020) method is proposed, which performs representation learning by contrasting two diffusion matrices transformed from the adjacency matrix. It reports better performance than variational GAE (VGAE) [Kipf and Welling 2016], marginalized GAE (MGAE) Wang et al. [2017] adversarially regularized GAE (ARGA) and VGAE (ARVGA) [Pan al., 2018 and GALA [Park]
et al. 2019 Different from MVGRL, our contrastive regularizer is directly applied on learned graph.
3 Methodology
3.1
Notation
Define the multi-view graph data as G sets of N nodes, Cij E E, denotes the relationship between node and node j in the v-th view, X (o,,...,.35)
Ei.., Ev,x",.... .Y., Xx,
where V represents the
the feature matrix. Adjacency matrices {A characterize the initial graph structure. {D Jomi represent the degree matrices in various views. The normalized adjacency matrix A = (D") (A" + ()(D))= and the corresponding graph laplacian I." = -- A
3.2 Graph Filtering
A feature matrix X E IR of N nodes can be treated as d N-ddimeniionll graph signals. A natural signal should be smooth on nearby nodes term of the underlying graph. The smoothed signals H can be achieved by solving the following optimization problem (Zhu et al. 2021 Lin et al. 2021b
min || H X + 8 Tr (H'LH).
H
where O is balance parameter and L is the laplacian matrix associated with X. H can be obtained by taking the derivative of Eq. (1) w.r.t. H and setting it to zero, which yields
H (I sL)
(2)
To get rid of matrix inversion, we approximate H by its first-order Taylor series expansion, i.e., H = (I - sL)X. Generally, m-th order graph filtering can be written as
(T - sL)"
where is non-negative integer. Graph filtering filter out undesirable high-frequency noise while preserving the graph geometric features.
3.3 Graph Learning
Since real-world graph is often noisy or incomplete, which will degrade the downstream task performance if it directly applied. Thus we learn an optimized graph S from the smoothed representation H. This can be realized based on the self-expression property of data, i.e., each data point can be represented by linear combination of other data samples (Lv et al.) [021] Ma al. 2020). And the combination coefficients represent the relationships among data points. The objective function on single-view data can be mathematically formulated as
min H -TS F
+ a ||

---

where S E RNN is the graph matrix and > 0 is the trade-off parameter. The first term is the reconstruction loss and the second term serves regularizer to avoid trivial solution. Many other regularizers could also be applied, such as the nuclear norm, sparse & norm [Kang et al. [2200a. To tackle vieti-view data, we can compute a smooth representation H2 for each view and extend Eq. (4) by introducing weighting factor to distinguish the contributions of different views
min S.A"
X" H
- -HTTT
2 + + + + (A")*,
(5)
where X" is the weight of v-th view and y is smooth parameter. Eq. (5) learns consensus graph S shared by all views. To learn a more discriminative S, we introduce a novel regularizer in this work.
3.4 Graph Contrastive Regularizer
Generally, contrastive learning is performed at instance-level and positive/negative pairs are con- structed by data augmentation. Most graph contrastive learning methods conduct random corruption on nodes and edges to learn a good node representation. Different from them, each node and its k-nearest neighbors (kNN) are regarded as positive pairs in this paper. Then, we perform contrastive learning at graph-level by applying contrastive regularizer on the graph matrix S instead of node features. can be expressed as
exp (Sij)
Σ log
Eppt exp (Sip)
JEN;
where N represents the k-nearest neighbors node w-th view. The introduction of Eq. (6) is to draw neighbors close and push non-neighbors apart, to boost the quality of graph. Eventually, our proposed multi-view contrastive graph clustering (MCGC) model can be formulated
min S,A"
A"
2 -
+
log
exp (Sij)
+
(A9)
(7)
Epzi exp (Sip)
D=1
1=1 JENE
V=1
Different from existing multi-view clustering methods, MCGC explores the holistic information from both multi-view attributes and multiple structural graphs. Furthermore, it constructs a consensus graph from the smooth signal rather than the raw data.
3.5
Optimization
There are two groups of variables in Eq. (7) and it's difficult solve them directly. optimize them, we adopt alternating optimization strategy, in which we update one variable and fix all others at each time.
Fix X", Update S
Because A is fixed, our objective function can be expressed as
exp (Siz)
min Σx" x" #=1
H T + Σ
log
(8)
Σ exp (Sip)
1=1
JEN
can be elemently solved by gradient descent and its derivative epoch can be denoted
(
+oV
The first term is
r" - [
(10)
1=1
Define (t-1)) Σ from each graph are all incorporated). Consequently, the second term becomes
Eppe
ppts exp (
and let n be the total number of neighbors (i.c., the neighbors
nexp (exp(((t-1) (((t-)))
Σ
-1
if Ny, otherwise.
R(e-1)
(4)
(11)
n eexp nexp(s(tt (((-11))
Σ
2
k(e-1)
r=1

---

Then we adopt Adam optimization strategy [Kingma and Ba, 2015 to update S. To increase the speed of convergence, we initialize with S where S is the solution of Eq. (5). Fix S, Update X" For each view v, we define M HoT -
S F + +aJ. Then, the loss function is simplified
V Σ
x"M" + Eary.
min
(12)
X e=1
By setting its derivation to zero, we get
M
X
(13)
We alternatively update S and X" until convergence. Algorithm 1.
The complete procedures are outlined in
Algorithm MCGC
Require: adjacency matrix A AV....A feature X² "....VVV the order of graph filtering m, parameter a, and y, the number of clusters c.
Ensure: c cluusters.
X" 1;
2: A" ( (Ā' +14++116 +D"""") :;
3: L" =I-
4: Graph filtering by Eq. (4) for each view;
5: while convergence condition does not meet do
6: Update 5 in Eq. (9) via Adam;
7: for each view do
8: Update X" in Eq. (13);
9: end for
10: end while
2 [ 12: Clustering C.
11: C
4 Experiments
4.1 Datasets and Metrics
We evaluate MCGC on five benchmark datasets, ACM. DBLP, IMDB Fan et al. 2020 Amazon photos and Amazon computers [Shchur et al.) 2018). The statistical information is shown in Table [ ACM: It is paper network from the ACM dataset. Node attribute features are the elements of bag-of-words representing of each paper's 's keywords. The two graphs are constructed by two types of relationships: "Co-Author" means that two papers written by the same author and "Co-Subject" suggests that they focus on the same filed.
DBLP: It is author network from the DBLP dataset. Node attribute features are the elements of a bag-of-words. representing of each author's keywords. Three graphs are derived from the relationships: "Co-Author", "Co-Conference" and "Co-Term". which indicate that two authors have worked together on papers, published papers at the same conference, and published papers with the same terms.
IMDB: It is a movie network from the IMDB dataset. Node attribute features correspond to elements of bag-of-words representing of each movie. The relationships of being acted by the same actor (Co-actor) and directed by the same director (Co-director) are exploited to construct two graphs. Amazon photos and Amazon computers: They are segments of the Amazon co-purchase network dataset, in which nodes represent goods and features of each good are bag-of-word of product reviews, the edges means that two goods are purchased together. have multi-view attributes, second feature matrix is constructed via cartesian product by following [Cheng et al. 2020].
We adopt four popular clustering metries: Accuracy (ACC), normalized Mutual Information (NMI), Adjusted Rand Index (ARI), Fl score.

---

Table 1: The statistical information of datasets.
Dataset
Nodes
Features
Graph and Edges
Clusters
Co-Subject (29,281)
ACM
3,025
1,830
Co-Author (2.210,761)
Co-Author (1 (11,113)
Co-Conference (5,000,495)
DBLP
4,057
334
Co-Term (6, (6,776,335)
Co-Actor (98,010)
IMDB
4,780
1,232
Co-Director (21,018)
745
Amazon photos
7,487
7,487
Co-Purchase(1 (1,ccha)e(1190433)) 19,043)
8
767
Amazon computers
13,381
13,381
Co-Purchassc245,778)
4.2
Experiment Setup
We compare MCGC with multi-view methods as well as single-view methods. LINE (Tagg et al. 2013] and GAE [iip and Welling 2016] have been chosen as representatives of single-view methods, and we report the best one among the results from all views. Compared multi-view clustering methods include: PMNE [Lu al., 2017), RMSC [Xia al., 2014), SwMC [Nie al.] [017]. PMNE and SwMC only use structural information while RMSC only exploits attribute features. PMNE uses three strategies to project multilayer network into a continuous vector space, so we select the best result. MCGC is also compared with other methods that not only explore attribute features but also structural information, i.e., O2MA and O2MAC [Fan al.] 2020]. MAGCN [Cheng al.. 2020]. In addition, MCGC compared with COMPLETER [Lin et al. [021a] and MVGRL [Hassani and Khasahmadi 2020] that conduct contrastive learning to learn a common representation shared across features of different views and multiple graphs respectively. For an unbiased comparison, part of the results from [Fan et al., 2020]. Since the neighbors of each node on different views could be different, also examine another strategy: only the shared neighbors in the contrastive loss term, N₂ u.= NJ. And our method with this approach is marked as MCGC* During experiments, k = 10 is used to select neighbors and 7 is fixed as -4 since we find that it has little influence to the result. According to parameter analysis, we set m 2, s 0.5, and tune a. All experiments are conducted on the same machine with the Intel(R) Core(TM) i7-8700 3. 3.20GHz CPU, two GeForce GTX 1080 Ti GPUs and 64GB RAM. The implementation of MCGC public available
copy
4.3 Results
All results are shown in Table 2 and Table 3 Compared with single-view method GAE, MCGC improves ACC by more than 9%. 4%, 19% ACM, DBLP, IMDB, respectively. Though using deep neural networks, GAE can not explore the complementarity of views. Compared with PMNE, the ACC, NMI, ARI, Fl are boosted by 16% 20%, 20%, 12% on average. With respect to LINE, RMSC, SwMC, the improvement is more significant. This can be attributed to the exploration of both feature and structure information in MCGC. Although O2MA, O2MAC, and MAGCN capture attributes and structure information, MCGC still outperforms them considerably. Specifically, MCGC improves O2MAC average by almost 6%. 9%. 1 11% ACC, NMI, F1, respectively. With respect to MAGCN, the improvement is more than 20% for all metrics. Compared with contrastive learning- based approaches, our improvement also impressive. In particular, compared with COMPLETER, the improvement is more than 30% on Amazon datasets, which illustrates that MCGC benefits from the graph structure information. MCGC also enhances the performance of MVGRL by 20%. By comparing the results of MCGC and MCGC3 we can see that the strategy of choosing neighbors does have impact on performance.
bbatss/ggtthbb.co///neeemmCCCCC

---

Table 2: Results on ACM. DBLP, IMDB.
Method
LINE
GAE
PMNE RMSC SwMC
O2MA
O2MAC
MCGC
MCGC
ACC
0.6479
0.8216 0.6936 0.6315 0.3831
0,888
0.9042
0.9147
0,9055
NMI
0.3941
0,4914 0.4648 0.3973
0.4709
0.6515
0.6923
0.7126
0.6823
ACM
ARI
0.3433
0,5444
0,4302 0.3312
0.0838
0.6987
0.7394
0.7627
0.7385
F1
0.6594
0.8225
0.6955
0.5746
0.018
0.8894
0.9053
0.9155
0.9062
ACC 0.8689
0.8859
0.7925
0.8994
0.3253
0.904
0.9074
0.9298
0.9162
NMI 0.6676
0.6925 0.5914 0.7111
0.019
0.7257
0.7287
0.8302
0.7490
DBLP
ARI 0.6988
0.741
0.5265
0.7647
0.0159
0.7705
0.778
0.7746
0.7995
0.8546 F1
0.8743
0.7966
0.8248
0.2808
0.8976
0.9013
0.9252
0.9112
ACC 0.4268
0.4298
0.4958
0.2702
0.2453
0.4697
0.4502
0.6182
0.6113
NMI
0.0031
0.0402
0.0359 0.3775
0.0023
0.0524
0.0421
0.1149
0.1225
IMDB
ARI
-0.009
0.0473
0.0366
0.0054
0.0017
0.0753
0.0564
0.1833
0.1811
F1 0.287 0.4062
0.3906 0.0018 0.3164 0.4229
0.1459
0.4401
0.4512
Table Results on Amazon photos and Amazon computers. The '-' means that the method raises out-of-memory problem.
Dataset
Amazon photos
Amazon computers
ACC
NMI
ARI
F1
ACC
NMI
ARI
F1
COMPLETER
0.3678
0.2606
0.0759
0.3067
0.2417
0.1562
0.0536
0.1601
MVGRL.
0.5054
0.4331
0.2379
0.4599
0.2450
0.1012
0.0553
0.1706
MAGCN
0.5167
0.3897
0.2401
0.4736
MCGC
0.7164
0.6154
0.4323
0.6864
0.5967
0.5317
0.3902
0.5204
5
Ablation Study
5.1 The Effect of Contrastive Loss
By employing the contrastive regularizer, our method pulls neighbors into the same cluster, which decreases intra-cluster variance. To see its effect, we replace I with Frobenius term, i.e. Eq. (5). As can be seen from Table 4 the performance falls precipitously without contrastive loss on all datasets. MCGC achieves ACC improvements by 16%, 8%, 5%, 12% on DBLP, ACM, IMDB, Amazon datasets, respectively. For other metrics, the contrastive regularizer also enhances performance significantly. Above facts validate that MCGC benefits from the graph contrastive loss.
Table 4: Results of MCGC without contrastive loss.
Datasets
ACM
DBLP
IMDB
Amazon photos
Amazon computers
MCGC
0.9147
0.9298
0.6182
0.7164
0.5967
ACC
MCGC w/o J
0.8334
0.7658
0.5636
0.5882
0.4662
MCGC
0.7126
0.8302
0.1149
0.6154
0.5317
NMI
MCGC w/o J
0.5264
0.4621
0.0707
0.5372
0.3988
MCGC
0.7627
0.7746
0.1833
0,4323
0.3902
ARI
MCGC w/o
0.5779
0.4949
0.1451
0.2640
0.1745
MCGC
0.9155
0.9252
0.4401
0.6864
0,5204
FI
MCGC w/o J
0.8313
0.7601
0.4444
0.5437
0.3678

---

Table 5: Results in various views of MCGC on ACM and Amazon photos. G1 and G2 denote the graphs in different views.
Dataset
ACM
Amazon photos
G X
G2, X
G1. G2, X
X
x², G
x',
ACC
0.9088
0.8152
0.9147
0.4433
0,6935
0.7164
NMI
0.6929
0.4656
0.7126
0.3519
0.5976
0.6154
ARI
0.7470
0.5229
0.7627
0.1572
0,4291
0.4323
0.9097
0.8184
0.9155
0.3675
0.6734
0.6864
5.2 The Effect of Multi- View Learning
To demonstrate the effect of multi-view learning, we evaluate the performance of the following single-view model
exp(S1t)
min - F a
log
(14)
N
JEN,
exp(Sip)
pyl
Taking ACM and Amazon photos as examples, report the clustering performance of various scenarios in Table 3 We can observe that the best performance is always achieved when all views are incorporated. addition, we also see that the performance varies for different views. This justifies the necessity of X" in Eq. (7). Therefore, it is beneficial to explore the complementarity of multi-view information.
5.3 The Effect of Graph Filtering
To understand the contribution of graph filtering, we conduct another group of experiments. Without graph filtering, our objective function becomes
exp (Siz)
min S.A"
X - F a 2
Σ log
Σ (A')2 (15)
Enys exp (Sip)
JENNY
We denote this model MCGC- The results of MCGC- are shown in Table 6 With respect to MCGC, ACC on ACM, DBLP, IMDB drops by 0.8%, 1.3%, 0.8%, respectively. This indicates that graph filtering makes positive impact on model. For other metrics, MCGC also outperforms MCGC- in most cases.
Table 6: The results of MCGC- (without graph filtering).
Dataset
ACM
DBLP
IMDB
MCGC
MCGC-
MCGC
MCGC-
MCGC
MCGC-
ACC
0.9147
0.9061
0.9298
0.9162
0.6182
0.6109
NMI
0.7126
0.6974
0.8302
0.7490
0.1149
0.1219
ARI
0.7627
0.7439
0.7746
0.7995
0.1833
0,
F1
0.9155
0.9057
0.9252
0.9112
0.4401
0.4509
6 Parameter Analysis
Firstly, two parameters and are applied in graph filtering. Taking ACM as an example, show their influence on performance by setting m [1,2,3, 4, 5]. [0.01, 0.1, 0.3, 0.5, 1,3, 5, 10]
in Fig. [ It can be seen that MCGC achieves reasonable performance for small m and

---

ACC
NMI
ARI
8
Figure I: Sensitivity analysis of parameters m and on ACM.
Therefore, 2 and 0.5 in all experiments. Afterwards, tune the trade-off parameter [10-3, 0.1, 10, 10 10 As shown in Fig. 2 our method is not sensitive to a, which enhances the practicality in real-world applications. In addition, we plot the objective variation of Eq. (7) in Fig.
As observed from this figure, our method converges quickly.
275
37
(b)
(c)
2801
1000
(c)
(f)
g
(h)
ADO
0.00
NMI
6.1
0.10
080
40
136
DUST E1
1000
002T
900
1000
0.
2:01
01
16
100
1003
(i)
(i)
(k)
(I)
Figure 2: Sensitivity analysis of parameter a on ACM (a-d), DBLP (e-h), IMDB (i-I).
xtof
10
0.5
8.5
1066
loss
l0s8
0.3
7.5
opoch
opoch
opoch
(a) ACM
(b) DBLP
(c) IMDB
Figure 3: The evolution of objective function.
Conclusion
Multi-view graph clustering is till at nascent stage with many challenges remained unsolved. In this paper, we propose novel method (MCGC) learn consensus graph by exploiting not only attribute content but also graph structure information. Particularly, graph filtering is introduced to

---

filter out noisy components and contrastive regularizer is employed to further enhance the quality of learned graph. Experimental results on multi-view attributed graph datasets have shown the superior performance of our method. This study demonstrates that it is possible for shallow model to beat deep learning methods facing the systematic use of complex deep neural networks. Graph learning crucial to more and more tasks and applications. Just like other methods that learn from data, brings the risk of learning biases and perpetuating them the form of decisions. Thus our method should be deployed with careful consideration of any potential underlying biases in the data. One potential limitation our approach is that it could take a lot of memory if the data contain too many nodes. This is because the size of learned graph is N x N. Research on large scale network is left for future work.
Acknowledgments and Disclosure of Funding
This paper was in part supported by the Natural Science Foundation of China ANos.61800045 U19A2059).
References
Maria Brbić and Ivica Kopriva. Multi-view low-rank sparse subspace clustering. Pattern Recognition, 73:247-258, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning. pages 1597-1607, PMLR, 2020a.
Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. Advances in Neural Information Processing Systems, 33, 2020b.
Jiafeng Cheng, Qianqian Wang, Zhiqiang Tao, Deyan Xie, and Quanxue Gao. Multi-view attribute graph convolution networks for clustering, IJCAL, 2020.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR'05), volume 1, pages $39-546. IEEE, 2005.
Shaohua Fan, Xiao Wang, Chuan Shi, Emiao Lu, Ken Lin, and Bai Wang. One2multi graph autoencoder for multi-view graph clustering. In Proceedings of The Web Conference 2020, pages 3070-3076, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '06), volume 2, pages 1735-1742. IEEE, 2006.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference Machine Learning, pages 4116-4126. PMLR, 2020.
Zhenyu Huang, Joey Tianyi Zhou, Xi Peng, Changqing Zhang, Hongyuan Zhu, and Jiancheng Lx. Multi-view spectral clustering network. In IJCAI, pages 2563-2560 2019,
Zhao Kang, Xiao Lu, Yiwei Lu, chong Peng, Wenyu Chen, and Zenglin Xu. Structure learning with similarity preserving. Neural Networks, 129:138-148, 2020a.
Zhao Kang, Haiqi Pan, Steven C.H. Hoi, and Zenglin Xu. Robust graph learning from noisy data. IEEE Transactions Cybermetics, 1113)1833338

---

Zhao Kang, Xinjia Zhao, Shi, chong Peng. Hongyuan Zhu, Joey Tianyi Zhou, Xi Peng, Wenyu Chen, and Zenglin Xu. Partition level multiview subspace clustering. Neural Networks, 122:279-288, 2020c.
Zhao Kang, Zhiping Lin, Xiaofeng Zhu, and Wenbo Xu. Structured graph learning for scalable subspace clustering: From single-view to multi-view. IEEE Transactions on Cybernetics, 2021. doi: 10.1109/TCYB.J. 1...222...66666
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Thomas Kipf and Max Welling. Variational graph auto-encoders. In NIPS Bayesian Deep Learning Workshop, 2016.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In AAAI 2021, volume 35, Feb. 2021.
Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng. Completer: Incomplete multi-view clustering via contrastive prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021a.
Zhiping Lin and Zhao Kang. Graph filter- based multi-view attributed graph clustering. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAL, pages 19-26, 2021.
Zhiping Lin, Zhao Kang, Lizong Zhang, and Ling Tian. Multi-view attributed graph clustering. IEEE Transactions Knowledge and Data Engineering, 2021b. doi: 10. 10.1109/TKDDE2221111111
Changshu Liu, Liangjian Wen, Zhao Kang, Guangchun Luo, and Ling Tian. Self-supervised consensus representation learning for attributed graph. In Proceedings of the 29th ACM International Conference Multimedia, 2021.
Weiyi Liu, Pin-Yu Chen, Sailung Yeung, Toyotaro Suzumura, and Lingli Chen. Principled multilayer network embedding. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pages 134-141. IEEE, 2017.
Juncheng Lv, Zhao Kang, Xiao Lu, and Zenglin Xu. Pseudo-supervised deep subspace clustering. IEEE Transactions on Image Processing, 30:5252-5266 2263,, 2021.
Zhengrui Ma, Zhao Kang, Guangchun Luo. Ling Tian, and Wenyu Chen. Towards clustering- friendly representations: Subspace clustering via graph filtering. In Proceedings of the 28th ACM International Conference Multimedia, pages 3081-3089, 2020.
Feiping Nie, Jing Li, Xuelong Li, al. Self-weighted multiview clustering with multiple graphs. In IJCAL, pages 2564-2570, 2017.
Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Adversarially regularized graph autoencoder for graph embedding. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2615, 2018.
Jiwoong Park, Minsik Lee, Hyung Jin Chang, Kyuewang Lee, and Jin Young Choi. Symmetric graph convolutional autoencoder for unsupervised graph representation learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6519-6528, 2019.
Xi Peng, Zhenyu Huang, Jiancheng Lv, Hongyuan Zhu, and Joey Tianyi Zhou. Comic: Multi-view clustering without parameter selection. In International Conference on Machine Learning, pages 5092-5101. PMLR, 2019.
Oleksandr Shchur. Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.0586 2018
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. Proceedings of the 24th international conference on world wide web, pages 1067-1077, 2015.

---

Chun Wang, Shirui Pan, Guodong Long, Xingquan Zhu, and Jing Jiang. Mgae: Marginalized graph autoencoder for graph clustering. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 889-898, 2017.
Hao Wang. Yan Yang, and Bing Liu. Gme: Graph-based multi-view clustering. IEEE Transactions on Knowledge and Data Engineering. 32(6):1116- 1129, 2019.
Rongkai Xia, Yan Pan, Lei Du, and Jian Yin. Robust multi-view spectral clustering via low-rank and sparse decomposition. In Proceedings of the AAAI conference artificial intelligence, volume 28, 2014.
Hongming Zhang, Liwei Qiu, Lingling Yi, and Yangqiu Song. Scalable multiplex network embedding. In IJCAI, volume 18. pages 3082-3088, 2018.
Huasong Zhong, Chong Chen, Zhongming Jin, and Xian-Sheng Hua. Deep robust clustering by contrastive learning. arXiv preprint arXiv:2008.030300 2020.
Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. Proceedings of the Web Conference 2021, pages 1226-1226, 2021.


Reviews:
[{"review_id": "pMZVqAaRNj_", "reviewer": "Reviewer_gTUK", "summary": "This paper proposes a generic framework for clustering multilayer/multi-view graph data. The main contributions include: \n- A graph filtering technique is developed for high-frequency noise removal;\n- A contrastive regularizer is applied to graph learning in a simple way; \n- Solid experiments and state-of-the-art performance evaluated on five graph data.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clarity: The paper is easy to follow and well organized.\n\nReproducibility: Yes, the source code is publicly available.\n\nStrengths：\n1. The challenge tackled by this paper seems interesting and the paper has a good motivation. To my best knowledge, the clustering of multi-view graphs is few studied and this is the first attempt to simultaneously handle multilayer graphs with multi-view attributes data.\n2. The proposed method is interesting and novel.  Instead of deep neural networks, the authors use graph filtering to obtain a good representation, which is simple and effective. Moreover, the idea of contrastive learning is employed in a new way to learn graphs in this paper. Though simple, the techniques make sense and look novel to me.\n3. The experimental evaluation is comprehensive and the results obtained with the proposed method are convincing. In particular, the authors compare many recent methods from different categories.\n4. This work could inspire more researchers/practitioners to apply a contrastive regularizer in shallow machine learning methods.\n\nWeaknesses：\n1. As shown in Table 1, the authors conduct experiments with multilayer graphs (ACM, DBLP, IMDB) and multi-view attributes (Amazon photos and computers) respectively. Is it possible to evaluate data with multilayer graph and multi-view features?\n2. The contributions of this paper should be explicitly highlighted in the Introduction.\n3. This paper presents abundant results with five tables. However, the discussion is quite short and should be elaborated in detail.\n4. The authors say that the cartesian product is applied to construct the 2nd feature. However, its definition is unclear.\n5. The authors should carefully proofread the whole paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitation and potential negative impact of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "j-J5FAG0uga", "reviewer": "Reviewer_p84e", "summary": "This paper proposes a multi-view clustering method. They first use graph filter to generate a new node representation for input, and then do self-expression in a linear combination of various views. Afterwards, they propose a graph contrastive regularizer, which mimics the contrastive learning given the nearest neighbors as positive samples and any others as negative ones. Experiments compared with recent works and ablation studies are presented.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The task is not new and the novelty seem lies in the contrastive regularizer.\n\nQuality: My major concerns are as follows.\n1. The graph contrastive regularizer, the main contribution of this work, is introduced without strong motivation or any deep insights and only takes up less than a half page. This makes the whole paper weak.\n2. From Eq. 6, the graph contrastive regularizer uses the neighbors information. That means this task is not a purely unsupervised task unless the neighbors are obtained from the distance of raw features. If so, from my view, this regularizer is close to spectral regularizer. If the authors insist that this new term contributes a lot, then they are supposed to add it on different optimization baselines to see the according improvements.\n3. From my understanding, Eq. 6 is similar to the contrastive loss in Raia 2006 if you simply reformulate the objective. Form this view, the improvement happens when the neighbors are reliable.\n4. From the alternative optimization, I can see the proposed method only do full batch update, which is not practical for real-world cases due to non-scalability. \n5. The simply used weights are not a good option for missing views or noisy views today. From this aspect, the authors over-claimed the impact of this work.\n(1) Raia et al., Dimensionality Reduction by Learning an Invariant Mapping, 2006\n\nClarity: The submission is clearly written.\n\nSignificance: Basically this work is not new for me and is lack of deep understanding. It does improve the current methods from the experimental results while the whole paper is weak.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I feel this work has some vital limitations like scalability for large dataset. They can use the matrix based method like years ago but the drawbacks of this kind of formulation should be mentioned and discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "Zo_DGAroUyU", "reviewer": "Reviewer_njqs", "summary": "This paper considers the multiview graph clustering problem where the goal is to find a cluster given multiple networks. The paper integrates the self-expression strategy and contrastive learning mechanism into a unified framework to perform graph learning. Besides, graph filtering is designed to recover a clean data in advance. An optimization algorithm is developed to solve the objective function. The paper compares the effectiveness of their method with other deep methods and demonstrates that the proposed method is competitive and promising.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Some strong points:\n1. Facing the systematic usage of complex deep learning, this paper makes an important contribution by showing that shallow model is still possible to beat deep neural networks-based methods. \n2. The method is simple to implement and easy to understand. The deployment of graph filtering is extremely interesting and could carry a lot of potential for future progress in the field. The combination of contrastive learning and traditional methods also has a lot of potential.\n3. Compared to the SOTA methods, the proposed method demonstrates its superiority on five benchmark datasets. Besides, many ablation experiments are carried out to the effectiveness of each component in the proposed objective function.\n4. The paper is well organized and easy to follow.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some concerns and questions:\n1. The complexity analysis is missing.\n2. Eq.(7) should be explained in more details. For example, it is unclear how to set $\\lambda$ in Eq.(7).\n3. It would be better to construct some other features and see how the proposed method will work.\n4. The paper needs a careful proofreading and some typos should be corrected, e.g. Eq.(11).", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "QubGqjehNfh", "reviewer": "Reviewer_NhsB", "summary": "This paper proposes a multi-view clustering model, named MCGC. It utilizes some modern techniques, such as graph filtering, graph learning and contrastive learning, to successfully deal with multiple graphs data with multi-view attributes.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main focus is on multiple graphs data with multi-view attributes, which is a lack for traditional multi-view clustering, and in this sense, MCGC is a more general framework compared with other SOTAs. It utilizes graph filtering to smooth node features, graph learning to discard noises and contrastive learning to optimize. And it outperforms other baselines to some degree.\n\nHowever, here are some drawbacks:\n1.\tthere is a big problem in the organization of the article. In introduction, the authors spend many efforts to describe previous multi-view clustering methods, which is confusing to readers, because the readers probably can not make full sense of these works. Meanwhile, the authors don’t state clearly about their motivation, for example why they should use contrastive learning. It’s difficult for me to follow the story. \n2.\tThe authors should survey some graph learning models in the part of related work.\n3.\tFor equation 1, I suggest the authors cite another paper “Interpreting and Unifying Graph Neural Networks with An Optimization Framework” [WWW 2021], which is also a related work.\n4.\tFor graph matrix S, it is initialized randomly, so its degree of freedom is very high. And the training of S is an inefficient process.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "2bFKxUtDcTl", "reviewer": "Reviewer_fzju", "summary": "propose a generic framework to cluster multi-view attributed graph data and borrow the contrastive learning idea to provide a  multi-view contrastive graph clustering (MCGC) method which outpeerforms SOTAs including deep methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Indeed, the submission has good readability and is easy to follow.  proposed MCGC is relatively intuitive and empirically-proved as well effective to some extent,  however, I think its novelty is NOT that significant, using the contrastive learning has had and NOT so unique. In addition, my comments are\n1. First there have existed many multi-view clustering (MVC) methods omitted here, secondly, major differences between the traditional MVC and the method for graphs here are unclearly made, especially what challenges will be posed or encountered.\n2. On lines 108-109, N-by-d feature matrix X is treated as d \"N-dimensional graph signals\", obviously, as the N increases, the graph size will be increased accordingly, which seems to bring an impracticality!\n3. On lines 115-117, A problem with (3) is if the L is defined by the feature matrix X whose noise will unavoidably propagate into the L, thus H is still noisy unless the L is robustly and jointly learned as well. However, (1) will fail to yield a robust L!\n4. On line 143, (7) is just a mixed objective of existing related methods! its optimization is realized via existing techniques.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have pointed out some biases from the data like other methods and remind needing care when deploying the method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on multiview graph representation learning, proposing a graph clustering model and demonstrating its usefulness against  a number of baselines. The reviewers felt the model was well motivated with clear contributions, though there were some concerns about presentations, namely on how the main contributions are presented. However, the reviewers were satisfied with the promised changes: I encourage the authors to do so as this will make the paper more generally clearer in terms of impact (as well as proofreading). Finally, the reviewers were generally excited to see strong performance of a shallow model, and I agree that this is an important result in the context of the currently dominant "bigger is better" models. Therefore, I recommend acceptance.
Decision: Accept (Poster)