Self-supervised Heterogeneous Graph Pre-training Based on Structural Clustering
Authors: Yaming Yang
Ziyu Guan
Zhe Wang
Wei Zhao
Cai Xu
Weigang Lu
Jianbin Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Graph Self-supervised Learning
Heterogeneous Information Networks
HGNN Pre-training

Abstract: Recent self-supervised pre-training methods on Heterogeneous Information Networks (HINs) have shown promising competitiveness over traditional semi-supervised Heterogeneous Graph Neural Networks (HGNNs). Unfortunately, their performance heavily depends on careful customization of various strategies for generating high-quality positive examples and negative examples, which notably limits their flexibility and generalization ability. In this work, we present SHGP, a novel Self-supervised Heterogeneous Graph Pre-training approach, which does not need to generate any positive examples or negative examples. It consists of two modules that share the same attention-aggregation scheme. In each iteration, the Att-LPA module produces pseudo-labels through structural clustering, which serve as the self-supervision signals to guide the Att-HGNN module to learn object embeddings and attention coefficients. The two modules can effectively utilize and enhance each other, promoting the model to learn discriminative embeddings. Extensive experiments on four real-world datasets demonstrate the superior effectiveness of SHGP against state-of-the-art unsupervised baselines and even semi-supervised baselines. We release our source code at: https://github.com/kepsail/SHGP.

Self-supervised Heterogeneous Graph Pre-training Based on Structural Clustering
Yaming Yang, Ziyu Guan, Zhe Wang, Wei Zhao; Cai Xu, Weigang Lu, Jianbin Huang School of Computer Science and Technology, Xidian University (yym@, zyguan@, zwang@ stu., ywzhao@ mail., cxu@, wglu@stu., jbhuange ]xidian.edu. edu.en
Abstract
Recent self-supervised pre-training methods on Heterogeneous Information Net- works (HINs) have shown promising competitiveness over traditional semi- supervised Heterogeneous Graph Neural Networks (HGNNs). Unfortunately, their performance heavily depends on careful customization of various strategies for gen- erating high-quality positive examples and negative examples, which notably limits their flexibility and generalization ability. In this work, we present SHGP, novel Self-supervised Heterogeneous Graph Pre-train Pre-training approach, which does not need generate any positive examples negative examples. It consists of two modules scheme. each iteration, the Att-LPA
that share the same iitentionnaggregaan module produces pscudo-labels through structural clustering, which serve the self-supervision signals to guide the Att- HG-N module to learn object embeddings and attention coefficients. The two modules effectively utilize and enhance each other, promoting the model to learn discriminative embeddings. Extensive experi- ments on four real-world datasets demonstrate the superior effectiveness of SHGP against state-of ofte--ff-hheart unsupervised baselines and even semi-supervised baselines. We release our source code https: htapp:///gtthin ///gggttubb.... ctmttuu.oom///// wwt..cccm/eeaaaee ksmp/aasssalSCC
Introduction
Over the past few years, various semi-supervised graph neural networks (GNNs) have been proposed to learn graph embeddings. They have achieved remarkable success in many graph analytic tasks. This success, however, comes at the cost a heavy reliance on high-quality supervision labels. In real-world scenarios, labels are usually expensive to acquire, and sometimes even impossible due to privacy concerns.
To relieve the label scarcity issue in (semi-) supervised learning, and take full advantage of large amount of easily available unlabeled data, the self-supervised learning (SSL) paradigm has recently drawn considerable research interest in the computer vision research community. It leverages the supervision signal from the data itself to learn generalizable embeddings, which are then transferred various downstream tasks with only few task-specific labels. One of the most common SSL paradigms is contrastive learning, which learns representations by estimating and maximizing the mutual information between the input and the output of deep neural network encoder g
40 have shown promising competitiveness compared with semi-supervised GNNs. They usually require three typical steps: (1) constructing positive examples (semantically correlated structural instances) by strategies such node dropping, edge perturbation, and negative examples (uncorrelated instances) by strategies such as feature shuffling, mini-batch sampling; (2) encoding these examples through graph encoders such as GCN [16]; (3) maxinizing/minimiiinn the similarity between these positive/negative
For graphs, some recent graph contrastive learning methods 22 7 41 25
36
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

examples. Nevertheless, in the real world, graphs often contain multiple types of objects and multiple types of relationships between them, which are called heterogeneous graphs, or heterogeneous information networks (HINs) 27 Due to the challenges caused by the heterogeneity, existing SSL methods on homogeneous graphs cannot be straightforwardly applied to HINs. Very recently, several works have made some efforts to conduct SSL on HINs (33 23 18 L3 14 [0]. In comparison with SSL methods on homogeneous graphs, the key difference is that they usually have different example generation strategies,
to capture the heterogeneous structural properties in HINs.
The strategies of generating high-qualit, positive/negative examples are critical the performance of existing methods [34 4 41 [6]. Unfortunately, whether for homogeneous graphs or heterogeneous graphs, the example generation strategies are dataset-specific. and may not be applicable all scenarios. This is because real-world graphs are abstractions of things from various domains, e.g., social networks, citation networks, etc. They usually have significantly different structural properties and semantics. Previous works have systematically studied this and found that different strategies are good at capturing different structural semantics. For example, study [41 observed that edge perturbation benefits social networks but hurts some biochemical networks, and study (36) observed that negative examples benefit sparser graphs. Consequently, in practice, the example generation strategies have to be empirically constructed and investigated through either trial-and-error or rules of thumb. This significantly limits the practicality and general applicability of existing methods.
In this work, we focus on HINs which are more challenging, and propose novel SSL approach, named SHGP. Different from existing methods, SHGP requires neither positive examples nor negative examples, thus circumventing the above issues. Specifically, SHGP adopts any HGNN model that is based on attention aaggiiggtgggggttionnnn scheme as the base encoder, which is termed as the module AIt-HGNN. The attention coefficients in Att-HGNN are particularly used to combine with the structural clustering method LPA (label propagation algorithm) [21]. as the module Att-LPA. Through performing structural clustering HINs, Att-LPA is able to produce clustering labels, which are treated as pseudo-labels. In turn, these pseudo-labels serve as guidance signals to help Att-HGNN learn better embeddings well better attention coefficients. Thus, the two modules are able to exploit and enhance each other, finally leading the model to learn discriminative and informative embeddings. In summary, have three main contributions follows:
We propose a novel SSL method on HINs, SHGP. It innovatively consists of the Att-LPA module and the Att-HGNN module. The two modules can effectively enhance each other, facilitating the model to learn effective emheddings.
To the best of our knowledge, SHGP the first attempt to perform SSL on HINs without any positive or negative examples. Therefore, it can directly avoid the laborious investigation of example generation strategies, improving the model's generalization ability and flexibility.
We transfer the object embeddings learned by SHGP various downstream tasks. The experimental results show that SHGP can outperform state-of-the-at basclines, even including some semi- supervised baselines, demonstrating its superior effectiveness.
2 Related work
14 10 that conduct SSL on HINs. Determined by their contrastive loss functions, all these methods require high-quality positive and negative examples to effectively learn embeddings. Thus, their effectiveness and performance hinge on the specific strategies of generating positive examples and negative examples, which limits their flexibility and generalization ability.
SSL on HINs. There are several existing methods 33 23 18
15
37 13
SSL on homogeneous graphs. Existing SSL methods on homogeneous graphs 29 20 19 43
7
35| 30 also need to generate sufficient positive and negative examples to effectively perform graph contrastive learning. They only handle homogeneous graphs and cannot be easily applied to HINs. In this work, seek to perform SSL on HINs without any positive examples or negative examples.
36
GNN+LPA methods. There exist several methods 31 However, they are all supervised learning methods, and only deal with homogeneous graphs. this work, study SSL on HINs.
24 that combine LPA 21 with GNNs.

---

Others. DeepCluster 2 uses K-means to perform clustering in the embedding space. Differently, our SHGP directly performs structural clustering in the graph space. JOAO [40] explores the automatic selection of positive example generation strategies, which is still not fully automatic. HuBERT [9] an SSI. approach for speech representation learning. GIANT [3] leverages graph- structured self-supervision to extract numerical node features from raw data. MARU 122 learns object embeddings by exploiting meta-contexts in random walks. Different from them, in this work, we study how effectively conduct SSL HINs. Graph pooling methods e.g. [22 39 learn soft cluster assignment to coarsen graph topology in each model layer. Differently, our method propagates integer (hard) cluster labels in each layer to perform structural clustering.
3 Preliminaries
We first briefly introduce some concepts about HINs, and then formally describe the problem we study in this paper.
Heterogeneous Information Network. An HIN is defined as: G = (V,E,A, R, Φ vb), where V the set of objects, & is the set of links, o: type mapping function and the link type mapping function, A denotes the set of object types, and R denotes the set of relations (link types), where [A| + [R] 2. Let X' (X,,... X AI denote the set containing all the feature matrices associated with each type of objects. A meta-path P of length
A and &
are respectively the object
I defined the form A₁ Ri Ag R2 describes a composite relation R = R1 R₂ o.... Rí between object types A1 and A441, where denotes the composition operator on relations.
Ri A(+1 (abbreviated as A A1 ... A(+1). which
We show toy HIN in the left part of Figure I It contains four object types: "Paper" (P), "Author" (A), "Conference" (C) and "Term" (T), and three relations: "Publish" between P and C, "Write" between P and A, and "Contain" between P and T. APC is meta-path of length two, and a1P202 is such a path instance, which means that author aj has published paper P2 in conference cz.
SSL on HINs. Given an HIN G, the problem is to learn an embedding vector h; E Rd for each object E in self-supervised manner, i.e., without using any task-specific labels. The pre-trained embeddings are expected to capture the general-purpose information contained in G, and can be easily transferred to various unseen downstream tasks with only few task-specific labels.
Conf
t
Att-LPA
Pozsdo labels
Term
CE loss
Paper
P1
P2
P
Author
AIt-HGNN
Alt line woight
Grad descent
Figure 1: The overall architecture of SHGP. Given an HIN, each iteration, we use Att-HGNN produce embeddings and predictions, and use At-ILPA produce pseudo-labels. The loss is computed as the cross-entropy between the predictions and the pseudo-labels. The attention coefficients (and other parameters) optimized via gradient descent, which attention-agget agggriattonnogttiooonnnn weights of Att-HGCN and Att-LPA in the next iteration, promoting them to produce better embeddings and predictions, as well better pseudo-labels.
the new
4 Methodology
In this section, we present the proposed method SHGP, which consists of two key modules. The Att- HGNN module instantiated as any HGNN model that based on attention-aggregatiion scheme. The Att-LPP module combines the structural clustering method IPA 21 with the attention-aggreegation scheme used Att-HGNN. The overall model architecture is shown in Figure Tand explained the figure caption. In the following, we describe the procedure of SHGP in detail.
3

---

4.1 Initialization
At the beginning. we randomly initialize all the model parameters in Att-HGNN by the Xavier uniform distribution [6]. To obtain initial pseudo-labels, we use the original LPA [2] to perform a thorough structural clustering on the input HIN G. Specifically, LPA randomly associates each object with unique integer its initial label, and lets them iteratively propagate along the links in G. In each iteration, each object updates After convergence, the final label indicates the cluster to which each object belongs. We treat these clustering labels returned by LPA as the initial pseudo-labels, and re-organize them one-hot label matrix Y R where K denotes the cluster size, which is not ppprr-paraeeter but depends on the uniqueness of these labels. Thus, in the subsequent steps, the propagation of the pseudo-labele labels can be easily implemented via the matrix multiplication operation.
label to the label that appears most frequently in its neighborhood.
Under the guidance of these obtained initial pseudo-labels, we first train Att-HGNN for several epochs the model "warm-up", to learn the initial meaningful attention coefficients as well other model parameters. We use y0(0) to denote all these initial parameters. Then, proceed with the following iterations.
4.2 Iteration
In the t-th iteration, we compute the object embeddings H( by the Att-HGNN module. Att-HGNN parameterized by W(t-11, and its inputs include: the HIN topology G, the object features X. This formulated follows:
H
Att-
(1)
Meanwhile, we update the pseudo-labels by the creatively proposed Att-LPA module. comparison with Att-HGNN, Att-LPA does not input X, but instead inputs y the pseudo-labels of the previous iteration. This is formulated follows:
Y Att-LPA (Wit-11, )
Att-HGNN and Att-LPA perform forward pass of attention-basee aggregation in the same way. The only difference between them that Att-HGNN aggregates the (projected) features of neighbors while Att-LPA aggregates the pseudo-labels of neighbors produced in the previous iteration, both weighted by exactly the same attention coefficients.
Now, we input (t) into softmax classifier which is built on the top layer of the Att-HGNN make predictions pit]_ The loss is computed as the cross-entropy between the predictions pll and the pseudo-labels Y follows:
pit =sofmmax [1) 10
Clt-1l)
c
(3)
1e0 40
iEV
where Cit-1) Wit-11 denotes the parameter matrix of the classifier. Having the loss, we can then optimize all the model parameters by gradient descent, as follows:
W = wit-11 1) V [(
(4)
where 7 denotes the learning rate. As the optimization proceeds, the model will learn better model parameters (including attention coefficients). This leads Att-HGNN and Att-LPA respectively produce better embeddings (and predictions) and better pseudo-labels in the next iteration, which, in turn, promotes the model to learn further better parameters. Thus, the two processes are able to closely interact with each other, as well as enhance each other, finally resulting in discriminative and informative embeddings. The overall procedure shown in Algorithm
4.3
Model discussion
4.3.1
Encoder of Att-HGNN
The AttHGGNN module can specifically instantiated as any attention-based HGNN encoders. Existing possible choices include: HAN 32). HGT [T] GTN 42), and ic-HGCN (38). Among them,

---

Algorithm The overall procedure of SHGP
Input: An HIN G
Output: Object embeddings
1: Perform a thorough LPA process to get initial lseudo-labels.
2: Guided by the initial pseudo-labels, warm-up Att-HGNN for several epochs.
3: while Not Converged do
4: Perform one forward pass of Att-HGNN module to compute embeddings by Eg.
1
5: Perform one forward pass of Att-LPA module to update pseudo-labels by Eq.
6: Compute cross-entropy loss by Eq. (3)
7: Optimize all the model parameters (including attention coefficients) by Eq. (4)
8: end while
our previously proposed ie-HGCN [8] is simple and efficient. It has shown superior performance the other three models. Therefore, in this work, adopt the base encoder of Att-HGNN, which is in detail described as follows.
Let Mr denote the set of object i's neighbors. Specially, object i itself is also added to Mi. Accordingly, an edge (i, i) added E, and a dummy self-relation (i, i) is added to R. each model layer, object i's new representation fi is computed as follows:
7 =
Σ JEN:
3 (ott))) 2i.) wo(.) hy 0i
(5)
where is the non-linear activation function, and hi neighbor j's current representation. We(.)) ((((i))) is the normalized link weight (2.0)) = 1), and g is
is the projection parameter matrix, 4.g the normalized attention coefficient, all of which are specific to relation (i,
With Att-HGNN, formulate our Att-LPA in similar way. Specifically, in each layer, object
i's pseudo-label is updated as follows:
Tel = one-hot
argmax
Σ
3 (
(6)
JEN;
where Vy is neighbor j's pseudo-label vector in the one-hot form. Object i first aggregates its neighbors' pseudo-labels according to the same normalized link weights and attention coefficients as in Eq. 5 Then, its new pseudo-label vector Th is obtained through the argmax operator and the one-hot encoding function.
The above two equations describe the computational details of one model layer, where the iteration indices and the layer indices are omitted for notation brevity. Here, we can further use the superscript (rr, y] to index symbol with respect z-th iteration and y-th layer. Assume the model has N layers. In the t-th iteration, for the input layer of Att-HGNN, we set: A...0 = H, (i.e., the object feature vector), while for Att-LPA, we set: y! [t,0) y ( At the end of the t-th iteration, properly
organize all the il! [[l..N] and (t.N as H [1 in Eq. (1) and y( in Eq. ) respectively. In this way, based on the ie-HGCN encoder, we establish our Au-- module and Att-LPA module.
4.3.2 Time complexity
As analyzed in the literature, both LPA 21 and ie- HG-H 38 have quasi-liness time complexity. Therefore, our SHGP also has quasi-linear time complexity, i.e., O(D) + [g)). Please see Appendix A.2 for the efficiency study of SHGP.
4.3.3 Label consistency
For SHGP, in the initialization phase, we perform the thorough LPA process until convergence to obtain the initial pseudo-labels, and warm-up Att-HGNN module to learn initial meaningful attention coefficients. each subsequent iteration, we no longer need to perform LPA from scratch, but instead perform one forward pass of Att- Therefore, the clustering results are basically consistent
5

---

with the results of the previous iteration. Thus, our SHGP does not need to perform alignment between clustering labels and predictions. This can avoid the issue DeepCluster [2] that there is no correspondence between two consecutive clustering assignments, and can also avoid the alignment process in M3S [26].
4.3.4 Class space
In this work, we cluster all types of objects into one common class space. We note that in most existing HIN datasets, various types of objects can indeed share the same class space. They typically show "star" network schema [28). since they are usually constructed according to one "hub" type of objects. The other types of objects are actually the attributes of the "hub" objects. For example, on DBLP, the "hub" objects are paper (P) objects. These papers and their associated authors, conferences, and etc. can all be categorized into four research areas: DM, IR, DB, and AI.
In the following extensive experiments, we demonstrate that this strategy works well on existing widely used HIN benchmark datasets. In future work, we will further investigate the problem of clustering different types of objects into different class spaces.
Experiments
In this section, we verify the generalization ability of the proposed SHGP by transferring the pre-trained object embeddings to various downstream tasks including object classification, object clustering, and embedding visualization.
5.1 Datasets
In the experiments, use four publicly available HIN benchmark datasets, which widely used in previous related works [38
33). Their statistics are summarized in Table [ Please see Appendix A.1 for more details of these datasets.
32
1
23
Table I: Dataset statistics.
Datasets
Objects (number)
Relations
MAG
P (4017), A (15383), I (1480), F (5454)
P P=R.P=E.==A--
ACM
P (4025), A (7167), S (60) A (4057), P (14328), C (20), T (8898) M (3328), A (42553), U (2103), D (2016)
P=AP-S
DBLP
P=AP-
IMDB
M=AM=UUM=D
MAG is subset of Microsoft Academic Graph. It contains four object types: Paper (P), Author (A), Institution (!) and Field (F)), and eight relations between them. Paper objects are labeled as four classes according to their published venues: IEEE Journal of Photovoltaics, Astrophysics, Low Temperature Physies, and Journal of Applied Meteorology and Climatology.
ACM is extracted from ACM digital library. It contains three object types: Paper (P), Author (A) and Subject (S), and four relations between them. Paper objects are divided into three classes: Data Mining, Database, and Computer Network.
DBLP extracted from DBLP bibliography. It contains four object types: Author (A), Paper (P), Conference (C) and Term (I), and six relations between them. Author (A) objects are labeled according to their four research areas: Data Mining, Information Retrieval, Database, and Artificial Intelligence.
IMDB is extracted from the online movie rating website IMDB. It contains four object types: Movie (M), Actor (A), User (U) and Director (D), and six relations between them. Movie (M) objects are categorized into four classes according to their genres; Comedy, Documentary, Drama, and Horror.
5.2 Baselines
We compare our SHGP with seven HIN-oriented baselines, including two semi-supervised methods, and five unsupervised suplf-supe) methods:

---

Semi-supervised: HAN
32 and ie- Ce-HGNN (abbreviated as HGCN) 38
Unsupervised: metapath2vee (abbreviated as M2V) [ DMGI 18 HDGI
23 HeCo
33
and
HGCN-DC (abbreviated as H-DC).
Here, HAN and HGCN semi-supervised HGNN methods. M2V is traditional unsupervised HIN embedding method. DMGI, HDGI, and HeCo are state-of-the-ar SSL methods on HINs. Note that, in the experiments, we adopt the baseline HGCN's encoder as the base encoder of Att-HGNN. H-DC is a variant of our SHGP. It also uses HGCN as the base encoder, but unlike SHGP which performs structural clustering in the graph space, H-DC uses K K-means to perform clustering in the embedding space, like DeepCluster [2]. We can use baselines HGCN and H-DC to investigate the effectiveness of the proposed supervuue pre-training scheme based on structural clustering.
5.3
Implementation details
For the proposed SHGP, in all the experiments, use two HGCN layers as the Att-HGNN encoder, and search the dimensionalities of the hidden layers in the set [64, 128, 256, 512). All the model parameters are initialized by the Xavier uniform distribution (6). and they are optimized through the Adam optimizer. The learning rate and weight decay are searched from le-4 to le-2. For the number of warm-up epochs, we search its best value the set {5, 10, 20, 30, 40, 50). We pre-train SHGP with up to 100 epochs and select the model with the lowest validation loss as the pre-trained model. Then, we freeze the model and transfer the learned object embeddings to various downstream tasks. For baselines,
reproduce their experimental results our datasets. Their syper-parameters are searched based on their papers and their official source codes. Some of the baseline methods B2
18 33 need users to input several meta- meta-paths, which are specified in Appendix A. in detail. For all the methods, we randomly repeat the evaluation tasks for ten times and report the average results. All the experiments are conducted on an NVIDIA GTX 1080Ti GPU.
5
Table 2: Object classification results (%).
Datasets
Metrics
Train
HAN
HGCN
M2V
DMGI
HDGI
HeCo
H-DC
SHGP
4%
90.07
93.16
88.97
94.43
94.10
95.75
85.03
98.23
Mic-FI
6%
91.83
95.18
89.94
93.80
93.68
95.93
85.16
98.30
8%
92.17
97.13
90.15
94.36
94.27
96.08
86.03
98.37
MAG
4%
89.93
92.82
88.51
94.32
93.89
95.27
84.72
98.24
Mac-Fl
6%
91.54
95.08
89.45
93.74
93.64
95.42
85.13
98.33
8%
91.82
97.05
89.73
94.27
94.23
95.15
85.97
98.41
4%
70.84
75.78
72.45
78.93
79.72
79.78
78.53
80.31
Mic-FI
6%
72.04
77.59
73.83
79.01
80.09
80.15
79.96
80.78
8%
73.23
78.08
73.95
79.47
79.07
80.94
79.82
80.91
ACM
4%
61.50
64.61
53.01
59.37
60.57
65.91
64.89
67.14
Mac-FI
6%
60.23
64.04
51.86
59.15
61.09
65.63
64.37
67.38
8%
62.37
65.73
53.72
59.42
59,99
67.15
65.11
68.19
4%
90.48
92.45
88.93
89.35
88.33
91.31
87.15
93.70
Mic-FI
6%
91.03
92.08
89.47
89.21
88.93
91.05
86.67
93.92
8%
91.90
92.34
91.41
89.88
88.18
91.22
87.23
94.13
DBLP
4%
90.01
92.13
88.49
88.21
87.69
90.53
87.03
93.31
Mac-FI
6%
90.51
91.71
88.97
88.03
88.75
90.26
86.53
93.52
8%
91.35
92.04
89.83
88.57
87.38
90.42
87.11
93.77
4%
56.05
56.68
56.54
54.79
56.31
57.42
54.01
58.51
Mic-Fl
6%
54.21
57.72
55.24
54.93
57.64
58.63
54.19
59.76
8%
56.45
57.03
57.02
55.75
56.70
60.13
55.19
61.60
IMDB
4%
39.04
36.66
27.03
37.95
30.84
38.66
34.72
43.36
Mac-Fi
6%
36.63
39.38
26.51
38.67
36.35
39.43
36.61
46.17
8%
38.20
40.54
27.86
39.89
34.64
40.00
38.03
48.02

---

5.4 Object classification
We conduct object classification on the object embeddings learned by all the semi semi- supervised baselines and unsupervised baselines. On each dataset, for the objects that have ground-truth labels, we randomly select {4%, 6%, 8%} objects as the training set. The others are divided equally as the validation set and the test set. For all the unsupervised methods, they don't use any class labels during learning object embeddings. After finishing the pre-training, the output object embeddings and their corresponding labels are used to train a linear logistic regression classifier. For all the semi-supervised methods, we directly report the classification results output by their own classifiers. We adopt Micro-FI and Macro-Fl as evaluation metrics. The results are reported in Table 2
We can see that our proposed SHGP achieves the best overall performance, even exceeding several semi-supervised learning methods, indicating its superior effectiveness. On MAG, SHGP achieves very high performance, i.e., over 98% Farro-F//MacroFF scores, which are close to saturation. Recall that SHGP adopts HGCN as the base encoder, and here, SHGP achieves better performance than HGCN. This indicates the effectiveness of the proposed strategy, i.e., pre-training HGNNs in a self-supervised manner based on structural clustering. SHGP also performs better than H-DC. This reasonable because H-DC uses K-means to perform clustering in the embedding space to obtain pseudo-labels, and thus it cannot effectively exploit the structural information which naturally resides in the graph space.
Most unsupervised pre-training methods outperform traditional semi-supervised methods. This verifies the superiority of the recent advances in the saf--supervised pre-training paradigm. Among the unsupervised methods, M2V performs much worse than the others. This is because M2V can only exploit a single meta-path, unlike DMGI, HDGI and HeCo which can well fuse the information conveyed by multiple meta-paths through regularization attention mechanism. Among the semi- supervised methods, HGCN outperforms HAN, probably because HAN can only exploit user-specified meta-paths while HGCN can automatically discover and exploit the most useful meta-patie (38)
5.5 Object clustering
We evaluate the embeddings through object clustering. In this task, we only consider the unsupervised methods, and the semi-supervised methods are excluded because they have exploited class labels during learning embeddings. On each dataset, we use K-means to cluster the embeddings of the labeled objects, and report normalized mutual information (NMI) and adjusted rand index (ARI) to quantitatively assess the clustering quality.
Table 3: Object clustering results (%).
MAG
ACM
DBLP
IMDB
NMI
ARI
NMI
ARI
NMI
ARI
NMI
ARI
M2V
39.67
43.75
32.53
28.49
49.50
56.73
1.43
1.03
DMGI
70.89
73.51
38.45
32.46
65.17
67.23
3.49
2.65
HDGI
73.96
77.15
39.13
32.34
59.98
62.33
4.15
2.96
HeCo
79.33
83.16
39.06
32.69
68.81
74.05
5.69
2.32
H-DC
42.75
49.01
18.60
19.75
47.15
53.15
1.57
1.12
SHGP
90.65
93.00
39.42
32.63
73.30
77.31
6.33
3.10
As shown in Table our SHGP achieves the best overall performance in this task. Especially, on MAG, SHGP outperforms other baselines by a large margin. This demonstrates its superior competitiveness against other baselines terms learning discriminative embeddings. On IMDB, all the methods have shown limited performance. This may be because the class labels and the structural information on this dataset are not significantly correlated. M2V shows the worst performance again, which may be due to its limitation as we analyzed in the previous experiment.
5.6 Visualization
For more intuitive comparison, on MAG, we visualize the paper objects' embeddings pre-trained by three unsupervised methods, i.c., DMGI, HeCo, and our SHGP. The embeddings are further
8

---

embedded into the 2-dimensional Euclidean space by the t-SNE algorithm 17 and they are colored according to their ground-truth labels.
(a) DMGI
(b) HeCo
(c) SHGP
Figure 2: Visualization of the pre- pre-trained embeddings of paper objects on MAG.
The results are shown in Figure We can see that DMGI shows blurred boundaries between different classes. HeCo performs relatively better than DMGI, but the green (left) class objects and the blue (bottom) class objects are still mixed some extent. Our SHGP shows the best within-class compactness and the clearest between-class boundaries, which demonstrates its superior effectiveness.
5.7 Hyper- parameter study
In this subsection, we investigate the sensitivity of the warm-up epochs, which is the main hyper- parameter that we have introduced in Section [.T] Specifically, on all the datasets, we explore the change of object classification performance the number of warm-up epochs gradually increases. The results are shown in Figure]
0.992
0.75
0.022
0.32
3.50
The Fum of worm-u apochs
The rum of womm-up spochs
The num of worm up spochs
The nmm warmisp epochs
(a) MAG
(b) ACM
(C) DBLP
(d) IMDB
Figure 3: Analysis of the number of warm-up epochs.
As shown, overall, this peree--araeeter not very sensitive. The general pattern is that the per- formance increases first, and then declines gradually. This is because, at the beginning, the model has not made full use of the information contained in the initial pseudo-labeh yet. After that, the model may gradually overfit these initial pseudo-labels, which prevents the model from continuously improving its performance in the iterations. The overall inflection points are 10, 20, 20, and 20 for MAG, ACM, DBLP, and IMDB respectively, which are the default values in the other experiments.
Conclusion
In this paper, we propose novel self-supervised pre-training method on HINs, named SHGP. It consists of two key modules which share the same attention -aggtionnagrgggationn scheme. The two modules are able to utilize and enhance each other, promoting the model to effectively learn informative embeddings. Different from existing SSL methods on HINs, our SHGP does not require any positive examples or negative examples, thereby enjoying a high degree of flexibility and ease of use. We conduct extensive experiments demonstrate the superior effectiveness of SHGP against state-of- the-art baselines.

---

Societal impact
Heterogeneous Information Networks (HINs) widely exist in our society, such as social networks, power networks, virus networks, etc. In this work, we develop a method to learn the representations for objects in HINs. The learned representations can be used for various analytical tasks such as object classification, object clustering, link prediction, etc. We believe that our method contributes to our society many aspects. However, there is also some risk that our method could be abused illegally or unethically for some undesirable purposes. We hope those bad things would not happen.
Funding transparency statement
This work was supported by the National Natural Science Foundation of China under Grants 62133012, 61936006, 62103314, 62203354, 61876144, 61876145, 62073255, 61876138 and 62002255, the Key Research and Development Program of Shaanxi under Grant 2020ZDLGY04-07, and the Innovation Capability Support Program of Shaanxi under Grant 2021TD-05.
References
[1] Claudio Bellei, Hussain Alattas, and Nesrine Kaaniche. Label-gen: An effective method for adding label propagation to graph convolutional networks. arXiv:2104.02155, 2021.
[2] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, pages 132-149, 149, 2018.
[3] Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovic, and Inderjit S Dhillon. Node feature extraction by self-supervised multi-scale neighborhood prediction. arXiv:2111.0000 1.0064, 2021.
[4] Ganqu Cui, Yufeng Du, Cheng Yang, Jie Zhou, Liang Xu, Lifeng Wang, and Zhiyuan Liu. Evaluating modules in graph contrastive learning. arXiv:2/06.08/71, 2021.
[5] Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. metapath2vec: Scalable representation learning for heterogeneous networks. In SIGKDD, pages 135-144. ACM, 2017.
[6] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pages 249-256, 2010.
[7] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-vies representation learning on graphs. In ICML, pages 4116-4126. PMLR, 2020.
[8] R Devon Hjelm, Alex Fedorov, Samuel Lavoic-Marchildoon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv:/808.0660 2018.
[9] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai,, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: self-supervised speech representation learning by masked prediction of hidden units. TASLP, 29:3451-3460 2021.
[10] Ziniu Hu, Yuxiao Dong, Kuansan Wang. Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In SIGKDD, pages 1857-1867, 2020.
[11] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In WWW, pages 2704-2710, 2020.
[12] Jyun-Yu Jiang, Zeyu Li, Chelsea J-T Ju, and Wei Wang. Maru: Meta-context aware random walks for heterogeneous network representation learning. In CIKM, pages 575-584, 2020.
[13] Xunqiang Jiang, Tianrui Jia, Yuan Fang, Chuan Shi, Zhe Lin, and Hui Wang. Pre-training on large-scale heterogeneous graph. In SIGKDD, pages 766,, 2021.
[14] Xunqiang Jiang, Yuanfu Lu, Yuan Fang, and Chuan Shi. Contrastive pre-training of gnns on heterogeneous graphs. In CIKM, pages 803-812, 2021.
10

---

[15] Baoyu Jing, Chanyoung Park, and Hanghang Tong. Hdmi: High-order deep multiplex infomax. In WWW. pages 2414-2424, 2021.
[16] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv:1609.020 02907, 2016.
[17] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(Nov): 2605, 2008.
[18] Chanyoung Park, Donghyun Kim, Jiawei Han, and Hwanjo Yu. Unsupervised attributed multiplex network embedding. In AAAI, pages 5371-5378, 2020.
[19] Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng. Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In WWW. pages 259-2700, 2020.
[20] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gec: Graph contrastive coding for graph neural network pre-training. In SIGKDDD, pages 1150-1160, 2020.
[21] Usha Nandini Raghavan, Réka Albert, and Soundar Kumara. Near linear time algorithm to detect community structures in large-scale networks. Physical review E, 76(3):036106 2007.
[22] Ekagra Ranjan, Soumya Sanyal, and Partha Talukdar. Asap: Adaptive structure aware pooling for learning hierarchical graph representations. In AAAI, pages 5470-5477, 2020.
[23] Yuxiang Ren, Bo Liu, Chao Huang, Peng Dai, Liefeng Bo, and Jiawei Zhang. Heterogeneous deep graph infomax. arXiv: 108588,,88888 2019.
[24] Yunsheng Shi, Zhengjie Huang, Wenjin Wang, Hui Zhong, Shikun Feng, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv:2009.03509 2020.
[25] Fan-Yun Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang, Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. arXiv:1908.01000, 2019.
[26] Ke Sun, Zhouchen Lin, and Zhanxing Zhu. Multi- state self-supervised learning for graph convolutional networks on graphs with few labeled nodes. In AAAI, pages 5892-5899, 2020.
[27] Yizhou Sun, Jiawei Han, Xifeng Yan, Philip S Yu, and Tianyi Wu. Pathsim: Meta path based top- k similarity search in heterogeneous information networks, VLDB Endowment, 4(11):992-1003, 2011.
[28] Yizhou Sun, Yintao Yu, and Jiawei Han. Ranking-basee based clustering of heterogeneous information networks with star network schema. In SIGKDD, pages 797-806, 2009.
[29] Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. 1rXiv:1809.10344 2018
[30] Haonan Wang, Jieyu Zhang, Qi Zhu, and Wei Huang. Augmentation-free graph contrastive learning, SSt1:22004.08874, 2022.
[31] Hongwei Wang and Jure Leskovec. Unifying graph convolutional neural networks and label propagation. arXiv:2002.06755: 2020.
[32] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang. Yanfang Ye, Peng Cui, and Philip S Yu. Heteroge- neous graph attention network. In WWW, pages 2022-2032, 2019.
[33] Xiao Wang, Nian Liu, Hui Han, and Chuan Shi. Self-supervised heterogeneous graph neural network with co-contrastive learning. In SIGKDD, page 1726-1736, 2021.
[34] Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Li. Self-supervised learning on graphs: Contrastive, generative, predictive. TKDE, 2021.
11

---

[35] Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. arXiv:2202.03104, 2022.
[36] Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, and Xiang Zhang. Infogel: Information-awwe awaae graph contrastive learning. arXiv:2110.15436 2021.
[37] Liang Yang, Fan Wu, Zichen Zheng, Bingxin Niu, Junhua Gu, Chuan Wang, Xiaochun Cao, and Yuanfang Guo. Heterogeneous graph information bottleneck. In IJCAI, pages 1638-168 1638-1645, 2021.
[38] Yaming Yang, Ziyu Guan, Jianxin Li, Wei Zhao, Jiangtao Cui, and Quan Wang. Interpretable and efficient heterogeneous graph convolutional network. TKDE, 2021.
[39] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. NeurlPS, 31, 2018.
[40] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. arXiv:2 2021.
[41] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. NeurIPS, 33:5812-58223, 2020.
[42] Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. In NeurlPS, pages 11960-11970, 2019.
[43] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW, pages 2069-2080, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section |
(b) Did you describe the limitations of your work? [Yes] See Section 4.3.4
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the text after Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read and ensured.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Abstract. (b) Did you specify all the training details (e.g., data splits, hyperparameters how they were chosen)? [Yes] See Section S the supplemental material, and our source code.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix A.3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5.3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We have cited the creators.
(b) Did you mention the license of the assets? [No] We only used publicly available assets.
(c) Did you include any new assets either the supplemental material or as URL.? [Yes]
We have released our source code. Please see Abstract for the URL.
12

---

(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [No] We only used publicly available assets.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] Our datasets are public and do not contain personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "yP3URi0cRa4", "reviewer": "Reviewer_1dEZ", "summary": "This paper studies the problem of learning object embeddings in heterogeneous information networks. To address this problem, the paper proposes a self-supervised graph learning model SHGP, which consists of two modules, namely Att-HGNN module and Att-LPA module. The main idea of the proposed method is to use the Att-LPA module to generate pseudo labels for objects, which are then leveraged to guide the Att-HGNN module to learn object embeddings. Experimental results on object classification and clustering verify the effectiveness of the method. ", "questions": "(1) How to choose the initial labels of objects in LPA module? Please see “strength and weakness” for more details about this question. \n(2) How to choose the number of clusters $K$ and how does this hyperparameter affects the performance of the proposed method?\n(3) Why use LPA to generate pseudo labels instead of other models such as HGNN?", "limitations": "The authors do not discuss the limitations and potential negative social impact of the proposed method in the paper. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of using structural clustering (label propagation method) to generate pseudo labels as self-supervision signals is novel and the experimental results verify the effectiveness of such a method. The paper is clear and easy to understand. \n\nHowever, there are some weaknesses in the paper:\n\n(1) Some technical details of the proposed method are unclear:\n\na) In section 4.1, the paper writes “LPA randomly associates each object with a unique integer as its initial label”. But where do these “unique integers” come from? Are they chosen from the ground-truth label set?\n\nb) How to determine the number of clusters $K$?\n\nc) In line 171, what is the link weight and how to obtain these weights? Similarly, how to obtain the attention coefficients? Are they model parameters?\n\n(2) The motivation of using LPA to generate pseudo labels is not clear. Why not use another HGNN to generate pseudo labels instead?\n\n(3) In the object classification experiment, I don’t think it is fair to use the object embeddings learned by semi-supervised baselines for comparison as they are not designed to learn object embeddings. Instead, I think the paper should compare the results of unsupervised methods with the results obtained by the classifiers of semi-supervised methods. \n\nMoreover, I am curious about how the number of clusters $K$ affects the performance of SHGP. \n\nFurthermore, the paper claims that the Att-HGNN module can use any attention-based HGNN encoders. However, the performance of the proposed method with different encoders is unknown. \n\n(4) The writing of the paper can be further improved:\n\na) I doubt whether the sentence in line 21 “This success, however, comes at the cost of a heavy reliance on high-quality supervision labels” is correct since most graph neural network models, such as GCN, GraphSAGE and GAT, are proposed to address the semi-supervised classification problem, which does not require a lot of labels. \n\nb) what do positive examples and negative examples mean? \n\nc) The full name of the proposed model SHGP is missing. \n\nd) I doubt whether the sentence “SHGP is the first attempt to perform SSL on HINs without any positive or negative examples” is correct since the paper does not explicitly explain what is positive examples and negative examples. Can we treat the ground-truth edges and object features as positive examples? If so, then the proposed model still uses positive examples. \n\ne) The citations of datasets are absent in the main paper. \n\nf) minors: \nIn line 99, conceptions —> concepts; \nIn line 235, punctuation is missing. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "yK8OrDsTTxo", "reviewer": "Reviewer_eMAZ", "summary": "This paper studies self-supervised learning (SSL) techniques for heterogeneous graph neural networks (HGNN). The proposed SSL method, SHGP, optimized attention-based HGNN layer (Attn-HGNN) by cross-entropy loss between encoder’s predictions and node’s pseudo cluster label obtained from the label propagation algorithm (Attn-LPA). In addition, SHGP demonstrated that iteratively training model parameters based on the updated pseudo labels from Attn-LPA can further improve the model performance.  ", "questions": "1. It is unclear which part of the structural clustering leads to the final good performance, as there are several hyper-parameters in pseudo node label cluster generation. For example,\n- How performance would change if we freeze (or only update only very few times) the pseudo node label, but just optimize the attn-HGCN layer?\n- How does the number of cluster sizes (K) affect the embedding quality? Can we also dynamically adjust the cluster size based on some coarse-to-fine heuristic? \n- If each node comes with side information (e.g., text description), can we cluster nodes based on label text representation?\nHow does the pseudo label assignment change between two attn-LPA updates? For example, how many nodes change their pseudo label cluster?\n\n2. Some implementation and model details are missing in Sec 5.3 and Table 1. For example,\n- Comparison model parameters between SHGP and baselines, which showed if the performance gain is due to larger encoder size and node embedding dimensions.\n- The pre-training time of SHGP and the baselines.\n", "limitations": "No potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n- The paper writing is clear and easy to follow\n- The empirical performance seems promising\n\n## Weakness\n- SHGP introduces several hyper-parameters, including the number of iterations for warm-up (line 2 in Algorithm 1), the number of iterations for main update (line 3 of Algorithm 1), the number of cluster size K.\n- Lack of comprehensive ablation study on structural clustering. See Questions section  for further comments. \n- Lack of proper citation for SSL based on dynamically-updated pseudo labels in the literature. For example, HuBERT (Hsu et al.) also considers pre-training tasks based on  pseudo labels, where k-mean clusters are dynamically updated.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1fSvyVhunc4", "reviewer": "Reviewer_ytZp", "summary": "In this paper, a self-supervised heterogeneous pre-training model (SHGP) is proposed. The Att-LPA module obtains node pseudo-labels through structured clustering, and the Att-HGNN module learns node embedding through attention mechanism. SHGP achieved significant results when compared with unsupervised or semi-supervised baseline methods.", "questions": "1. Is it necessary to use attention in both ATT-LPA and ATT-HGNN? Ablation study are required.\n2. How to overcome the error caused by randomness of LPA algorithm.\n3. In Line 5 of algorithm 1, will it be difficult for algorithm to converge if pseudo-labels are continuously updated?\n", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. In this paper, a new paradigm of self-supervised pre-training model on heterogeneous graphs is proposed, which generates pseudo-labels of nodes to guide the training of node embedding.\n2. In this paper, pseudo-labels are obtained through label propagation, avoiding the generation of positive or negative examples, thus improving the generalization of the model.\n\nWeaknesses:\n1. Pseudo-labels initialized by structural clustering only consider structural features and ignore important information in node features.\n2. The randomness of pseudo labels generated by LPA algorithm is large.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-TDCnzNFh5L", "reviewer": "Reviewer_cQLi", "summary": "In this paper, the authors propose to leverage structural clustering for pre-training heterogeneous graph neural networks in a self-supervised manner. The proposed SHGP framework adopts Attentive HGNN as the base encoder and utilizes label propagation for structural clustering, thereby deriving pseudo labels for  self-supervised pre-training. The experiments are conducted on several benchmark datasets of heterogeneous graphs. The experimental results show that the proposed SHGP outperforms several conventional baseline methods in both tasks of object classification and clustering.", "questions": "* The concept of considering clustering for graph neural networks is very similar to the concept of pooling. For example, [c,d] also learn a soft cluster assignment for enhancing model learning. The authors should compare with them and emphasize the motivation of the paper.\n\n    * [c] Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., & Leskovec, J. (2018). Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31.\n    * [d] Ranjan, E., Sanyal, S., & Talukdar, P. (2020, April). Asap: Adaptive structure aware pooling for learning hierarchical graph representations. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 5470-5477).\n\n* The whole framework combines two established components from other studies, but the key idea is still about structural clustering that is unnecessarily relevant to heterogeneous graphs. I wonder if the idea can be adopted with models for homogeneous graphs.\n\n*Some state-of-the-art pertaining methods for both general and heterogeneous graphs [a, b], as well as some unsupervised representation learning methods [c], are not discussed and compared in the experiments. It would be better to compare with the state-of-the art methods in the experiments.", "limitations": "The authors do not mention any limitations and potential negative societal impact of the work. However, graph neural net related algorithms could still have some risks about resulting some abuse scenarios on social networks.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Simple but effective framework.\n* Publicly available benchmark datasets and released implementations for the reproducibility. \n\n\nWeaknesses\n* The idea of using clustering and pooling to benefit graph neural nets is not novel. \n* Do not propose novel methods directly related to heterogeneous graphs.\n* Some state-of-the-art pertaining methods for both general and heterogeneous graphs [a, b], as well as some unsupervised representation learning methods [c], are not discussed and compared in the experiments.\n\n    * [a] Chien, E., Chang, W. C., Hsieh, C. J., Yu, H. F., Zhang, J., Milenkovic, O., & Dhillon, I. S. (2021, September). Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction. In International Conference on Learning Representations.\n    * [b] Wang, X., Liu, N., Han, H., & Shi, C. (2021, August). Self-supervised heterogeneous graph neural network with co-contrastive learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (pp. 1726-1736).\n    * [c] Jiang, J. Y., Li, Z., Ju, C. J. T., & Wang, W. (2020, October). Maru: Meta-context aware random walks for heterogeneous network representation learning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 575-584).\n\n* No theoretical supports.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept