Self-Supervised Heterogeneous Graph Learning:  a Homophily and Heterogeneity View
Authors: Yujie Mo
Feiping Nie
Ping Hu
Heng Tao Shen
Zheng Zhang
Xinchao Wang
Xiaofeng Zhu
Venue: ICLR 2024 poster (2024)
Keywords: Graph representation learning
Heterogeneous graph
Self-supervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper makes the first attempt to understand self-supervised heterogeneous graph learning without pre-defined meta-paths from the perspective of homophily and heterogeneity.

Abstract: Self-supervised heterogeneous graph learning has achieved promising results in various real applications, but it still suffers from the following issues: (i)  meta-paths can be employed to capture the homophily in the heterogeneous graph, but meta-paths are human-defined, requiring substantial expert knowledge and computational costs; and (ii) the heterogeneity in the heterogeneous graph is usually underutilized, leading to the loss of task-related information. To solve these issues, this paper proposes to capture both homophily and  heterogeneity in the heterogeneous graph without pre-defined meta-paths. Specifically, we propose to learn a self-expressive matrix to capture the homophily from the subspace and nearby neighbors. Meanwhile, we propose to capture the heterogeneity by aggregating the information of nodes from different types. We further design a consistency loss and a specificity loss, respectively, to extract the consistent information between homophily and heterogeneity and to preserve their specific task-related information. We theoretically analyze that the learned homophilous representations exhibit the grouping effect to capture the homophily, and considering both homophily and heterogeneity introduces more task-related information. Extensive experimental results verify the superiority of the proposed method on different downstream tasks.

Published as a conference paper at ICLR 2024
SELF- SELF-SUPERVISED HETEROGENEOUS GRAPH LEARN- ING: A HOMOPHILY AND HETEROGENEITY VIEW
Yujie Mo
Feiping Nie
Ping Hu
Heng Tao Shen
Zheng Zhang 2+ Xinchao Wang Xiaofeng Zhu
SSchool of Computer Science and Engineering. University of Electronic Science and Technology of China National University of Singapore
3 School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University Schooo of Computer Science and Technology, Harbin Institute of Technology, Shenzhen
ABSTRACT
Self-supervised heterogeneous graph learning has achieved promising results in various real applications, but it still suffers from the following issues: (i) meta- paths can be employed to capture the homophily in the heterogeneous graph, but meta-paths are human-defined, requiring substantial expert knowledge and com- putational costs; and (ii) the heterogeneity in the heterogeneous graph is usually underutilized, leading to the loss task-related information. To solve these issues, this paper proposes to capture both homophily and heterogeneity in the hetero- geneous graph without pre-defined meta meta-paths. Specifically, we propose to learn self-expressive matrix capture the homophily from the subspace and nearby neighbors. Meanwhile, we propose to capture the heterogeneity by aggregating the information of nodes from different types. We further design a consistency loss and a specificity loss, respectively, to extract the consistent information between homophily and heterogeneity and to preserve their specific task task-related information. We theoretically analyze that the learned homophilous representations exhibit the grouping effect to capture the homophily, and considering both homophily and heterogeneity introduces more task-related information. Extensive experimental results verify the superiority the proposed method on different downstream tasks.
INTRODUCTION
Heterogeneous graph learning aims to extract and uncover meaningful hidden patterns in the hetero- geneous graph, such that it outputs discriminative representations for different tasks (Dong et al.) 2017. Sun et al.) 2021). To alleviate the issue of limited labels in real scenarios, self-supervised heterogeneous graph learning (SHGL) has received increasing attention across diverse applications, such social network analysis and recommendation systems (Chen et al. 2022 Xie al. 2022).
Existing SHGL methods typically employ meta-paths to extract semantic relationships among nodes of the same type in the heterogeneous graph, as illustrated in Figure [[a] Consequently, such process treats the heterogeneous graph as a composition of homogeneous graphs based on meta- paths (Wang et al., 2023 Mo al. (023a). This actually mines the homophily (i.e., connectivity and information aggregation among nodes within the same class) in the heterogeneous graph, as two nodes connected by meta-path generally tend to belong to the same class. For instance, in an academic heterogeneous graph with several node types (e.g., author, paper, and subject), if there exists a meta-path saaer--athhorpppprr between two papers (i.e., two papers are written by the same author), these two papers possibly belong to the same class. As result, previous SHGL methods utilize meta-paths to explore the homophily, increasing the intra-class correlation and benefiting downstream tasks, as shown in Figure ((b)
"Corresponding authors: X. 7hu, X. Wang, and 7. Zhang

---

Published as a conference paper at ICLR 2024
Author (A) Paper (P) Subject (S)
Meta-path:
10
76-
50
40
20-
pulish
Mela-path:
written
published
Epoda
Mctapah-harcd mranh
(a) Example of meta- sitta-ppth--ie based graphs
(b) Studies of different meta meta-path sata--pahh--e path-based graphs
Figure 1: Example and studies of meta-path setta-path-baseee based graphs in previous SHGL. (a) For academic heterogeneous graph, most previous SHGL employs meta-paths (e.g. P-A-P and P-S-P) to establish connections between two papers, and then ignores nodes from other types (e.g.. Author) in meta- meta-paths. (b) Intra-class correlation and node classification results (i.e., Micro-F1) by GCN (Kipf & Welling 2017) meta-path-basee homogeneous graphs with different homophily ratios (HR, i.e. the ratio of nodes connected by meta-path belong to the same class). The higher the HR of the meta-path-based graph, the higher the intra-class correlation, thus benefiting the classification performance.
However, existing SHGL methods still have limitations that need to be addressed. On the one hand, meta-paths are manually defined and require expert knowledge to select appropriate meta- paths for different tasks (Lv et al. 2021). Moreover, employing meta-paths to extract the relationships among nodes incurs considerable computation costs, which exponentially increase with the meta-path length. On the other hand, most previous SHGL methods overlook cannot effectively utilize the heterogeneity (i.e., connectivity and information aggregation among nodes from different types) in the heterogeneous graph, which may carry significant information relevant to downstream tasks. Take the same academic heterogeneous graph as an example. If two authors have the same name in the same institution, difficult distinguish these two authors. In contrast, if we consider the heterogeneity (e.g., each author's 's published papers), we can easily distinguish them. As result, previous SHGL methods may lose significant task-related information associated with the heterogeneity.
Based on the above analysis, it is feasible to consider both homophily and heterogeneity in the heterogeneous graph without pre-defined meta-paths to improve the effectiveness of SHGL. To achieve this, there are at least two challenges to be solved, Le., (i) capturing the homophily in the heterogeneous graph without relying on meta-paths; and (ii) effectively utilizing both homophily and heterogeneity in the heterogeneous graph, despite their inherent conflict.
In this paper, to address the above challenges, we discard traditional meta- meta-paths and propose a novel SHGL framework to capture both Homophily and hetEROgeneity in the heterogeneous graph (HERO for short), as shown in Figure Specificallyy, we obtain the closed-form solution of the self-expressive matrix to capture the homophily from the subspace and nearby neighbors and obtain homophilous representations, thus tackling Challenge (i). Meanwhile, we employ a heterogeneous encoder to aggregate the information of nodes from different types to capture the heterogeneity and thereby obtain heterogeneous representations. With homophilous and heterogeneous representations, we further design consistency loss and specificity loss to capture the invariance between them and preserve their respective task-related information in the latent space, respectively, thus tackling Challenge (ii). Finally, in theoretical analysis, the learned homophilous representations are proved to capture the homophily in the heterogeneous graph, while homophilous and heterogeneous representations are fused introduce more information related to downstream tasks.
Compared to previous SHGL methods, our main contributions can be summarized as follows:
We make the first attempt understand the self-supervised heterogeneous graph learning without pre-defined meta-paths from the view of homophily and heterogeneity.
We propose to comprehensively capture the homophily from both the subspace and nearby neighbors as well as discard pre-defined meta-paths that require expert knowledge. We further extract consistent and specific information between homophilous and heterogeneous representations introduce more raskerelated information, thus achieving effectiveness.
2

---

Published as a conference paper at ICLR 2024
Projection
mulion
syer Passptinn B
Matmal
Projection
H
K
Self-expressive matrix
Projection
kk
litergmmana Enaader
P
Projection
fa
Figure 2: The flowchart of the proposed HERO. Specifically, HERO first employs the Multi-Layer Perception as encoder Os and learns a self-expressive matrix S to capture the homophily and obtain homophilous representations Z. Meanwhile, HERO employs heterogeneous encoder So to aggregate the information of nodes from different types to obtain heterogeneous representations Z. After that, HERO designs a consistency loss Lcon and specificity loss Lype to extract the consensus between Z and Z well as maintain their distinct information in different latent spaces, respectively.
We theoretically demonstrate that the learned homophilous representations have the grouping effect, thus capturing the homophily. Furthermore, we theoretically demonstrate that considering both homophily and heterogeneity introduces more task-related information than considering them individually, thus benefiting downstream tasks.
We experimentally demonstrate the superiority of the proposed method in terms of different downstream tasks on both heterogeneous graph datasets and homogeneous graph datasets, compared to numerous comparison methods.
METHOD
Notations. Let G
and represent heterogeneous graph, where V = [ N
indicate nodes set and edges set, respectively, and indicates the number of nodes. X = [x denotes the node features matrix, while T and R indicate node types set and edge types set, respectively. Given the heterogeneous graph, the meta-path used previous SHGL methods can defined in the form of 01 4 02 3 .... is Us+1- It is a sequence of a composite relation T1 o 12 rs between node U1 and node 1s+1. where indicates the length of meta-path and denotes the composition operator. Many previous SHGL methods explore the homophily in the heterogeneous graph with different meta-paths, shown in Figure [(a] In contrast, the proposed method mines both homophily and heterogeneity in the heterogeneous graph without pre-defined meta-paths, shown in Figure 2 and we introduce the details of the proposed method as follows.
2.1 HOMOPHILY
In this paper, we propose to adaptively leam the homophily in the heterogeneous graph without meta-paths. Actually, the homophily extraction in the heterogeneous graph aims establishing connections and conducting information aggregation among nodes within the same class, Considering
the learning scenario without labels, previous studies (Chapelle et al. 2002 Zhou et al. 2003 show two statements: (i) nodes in the same subspace are likely to belong to the same class; and (ii) nearby nodes are likely to belong to the same class. Hence, there are at least two ways for the homophily extraction in the heterogeneous graph. The first way is to connect and aggregate the information of nodes in the same subspace, while the second way is to connect and aggregate the information of each node and its nearby neighbors.

---

Published as a conference paper at ICLR 2024
In this paper, we first employ the Multi-Layer Perceptron (MLP) as the encoder J& to obtain the (1+1)-th layer node representations H(4+) by:
= (H w()),
(I)
where is the activation function, w indicates the trainable parameters of go. and H X. After that, we propose to capture the homophily in the subspace with a self-expressive matrix RNN that linearly describes every node by all nodes, i.e.
H (++)) SH(+1) + ((+1),
where O(++1) is a noise matrix. In Eq.
2 the representation of the i-th node h (1+1) can be (1+1) In particular, the larger the weight (i.e, the sy w:
represented by h (n(+t)) sq h ((+1))
value of $ty), the higher the probability of the node v; replaced by the node Vy. According to the teessacce-rreervvnn
property (He et al.) [2003] Vidal (2009). each node and its corresponding nodes with large weights are likely to fall in the same subspace, so they are likely to belong to the same class. Therefore, the self-expressive matrix describes each node by the nodes (i.e., with large weights) in the same subspace to capture the homophily.
However, the self-expressive matrix may also introduce the nodes from different classes (i.e., with small weights) into the subspace to degrade the model performance. A good solution is to push the weight of nodes from different classes to be as small as possible. Based on the second statement, we propose encourage the self- expressxees matrix to focus more on the neighbors of each node in the original feature space and less on its faraway nodes that may come from different classes. To do this, first calculate the feature distance matrix D T between all node pairs, where d. = |x; I1.. and then treat the value less than the threshold as to obtain sparse feature distance matrix. Based on the above analysis, each node v, and its neighbors set Mf (Vey E Nr, dy 0) should be further assigned with large weights in the self-expressive matrix while the weights of faraway nodes should be penalized. To achieve this, we propose to capture the homophily from both the subspace and nearby neighbors by:
N min H (++ SHO +a Σ 1,j=1
N
di
+
3
(,==1
where a and | are non-negative parameters trade off three terms. In Eq. (3). the second term enables the self-expressive matrix to focus on the nearby neighbors (i.e., with small feature distance) of each node to capture the homophily. Moreover, the second term penalizes the weights among nodes from different classes induced by the first term, and the first term captures part connections within the same class missed by the second term, thus complementing each other. The third term aims to regularize the self-expressive matrix S to avoid the trivial solution. Actually, Eq. (3) takes the closed-form solution S as follows:
S" = (1+1) (H"+ a (H (H+))) (H + +IN)
(4)
where I is the identity matrix. fact, the self-expressive matrix shares a similar idea with the widely used self-attention mechanism (Vaswani et al. (017). i.e., linearly describe each sample with all samples. However, there are major differences between them, and we list them in Appendix A.3
After achieving the closed-form solution S*, we conduct information aggregation among nodes of the same type in the heterogeneous graph to obtain homophilous representations Z, i.e.,
(5) However, directly obtaining Z by Eq. incurs expensive computation costs due to the cubic time complexity in computing S* in Eq. (4) and the quadratic time complexity in calculating S ((+1). To alleviate this issue, we avoid directly calculating S and reformulate Eq. (5) by the matrix identity transformation (Woodbury 1950) (details are shown in Appendix C.3 i.e.
Z = ('+1)
Z = H ((+)) H + DB,
(6)
where B 1 H(+ Ba
H (I. (H(+))) T (H ( H(+)) and I, E dddd the identity matrix, where d indicates the dimension of node representations. By reordering the
4

---

Published as a conference paper at ICLR 2024
matrix multiplication, further reduce the time complexity of Eq. 6 to O(N + d + kdl,, where d? < N and & < 222 and & indicates the nonzero entries of the sparse feature distance matrix D, details are shown in Appendix [8.2]
Therefore, the proposed method is available to capture the homophily in the heterogeneous graph in an effective and efficient way. In this paper, we further prove that both the self-expressive matrix S and the learned homophilous representations Z capture the homophily by having the grouping effect. To do this, we first follow et al. 2020) define the grouping effect follows.
Definition 2.1. Given the nodes set V {u if cjal + 0 (V1 S F') holds for every 2/ and Uy satisfying v, - 1 (i.e., ||x; Xy 112 + 0), the matrix RNF'' has the grouping effect.
Based on Definition 2.1 if a matrix C has the grouping effect and the condition v; - 2 holds for two nodes t; and vy then every element of the i-th and the j-th row (c, and cj) should be aligned. Indeed, the condition v, - v indicates two nodes may belong to the same class, thus the alignment between C₁ and Cj reflects the homophily among nodes. After that, we follow Definition 2.1 to prove the grouping effect of the self-expressive matrix S and homophilous representations Z by Theorem 2.2 whose proof can be found in Appendix C.4
Theorem 2.2. Both self- expressive matrix S E NNN and homophilous representations Z E Nxd have the grouping effect for any two nodes and Vi that hold the condition v; - Ui i.e.,
U/ +
sp
Spp + O, and Rig 23g + 0, < a d. iij.s.pps N.
(7)
Based on Theorem 2.2 if two nodes o and by have similar node features, i.e., being likely belong to the same class, both their self-expressive vectors (i.e., st and s;) and representations (i.e., and 2,)) are expected to be similar. As result, both S* and have the grouping effect, thus capturing the homophily in the heterogeneous graph (verified in Section 3.2.3
2.2 HETEROGENBITY
In addition to the homophily, the heterogeneity is also significant for the heterogeneous graph as it may contain task-related information Zhang et al.] (2022a). However, most existing SHGL methods overlook or cannot effectively utilize the heterogeneity from nodes of different types. As result, previous SHGL. methods may lose discriminative information the heterogeneity induce a negative impact on downstream tasks. Therefore, it is necessary to capture the heterogeneity in the heterogeneous graph to improve the effectiveness of SHGL.
To do this, we propose to aggregate the information of nodes from different types in the heterogeneous graph. Specifically, for the node v, we employ a heterogeneous encoder Sa to aggregate the information of its relevant one-hop neighbors (i.e., nodes of other types) based on the edge type E R, and then obtain the edge-based representations (((+1) by:
(1+1) -
5 ((
Σ
172
(1) Uy E NN1r) )
2
77
where indicates the activation function, Marr indicates the one-hop neighbors set of the node e; based on the edge type r, is the number of the neighbors, and w:!)) indicates the trainable parameters of fe. Considering all edge types in the heterogencous graph, we further obtain the heterogencous representations by fusing all edge-based representations, i.e..
(141)
1
Σ Z(+1)
|R|
PR
Z represent the last layer of heterogencous representations (44 for brief, where L is the number of layers. As a result, the heterogeneous representations aggregate the information of nodes from different types and thus are expected to capture the heterogeneity in the heterogeneous graph.
where |R] indicates the number of edge types. Finally,
5

---

Published as a conference paper at ICLR 2024
2.3 CONNECTIONS BETWEEN HOMOPHILY AND HETEROGENBITY
Given homophilous representations Z and heterogeneous representations Z. we have the following observations: (i) they are both representations of the same node, sharing the same original node features. Therefore, they are intuitive to contain the consistent information; and (ii) the homophilous representations focus on aggregating the information from nodes of the same class, while the heterogeneous representations focus on aggregating the information from nodes of different types. As a result, homophilous and heterogeneous representations contain specific information within each of them, respectively. To effectively utilize homophilous and heterogeneous representations (i.e., Z and Z). we design a consistency loss and specificity loss to extract the consistent information between them and to maintain their individually specific information related to downstream tasks, respectively.
Specifically, we first propose learn projection head Pe to map both homophilous representations and heterogeneous representations into the same latent space, i.e., P Pe (Z) and P = (Z). After that, we design consistency loss to maximize the invariance between P and P by:
N
Leon Σ (px 2=) + log(
(10)
1=1
JJ=1
where and j indicate i-th and j-th dimensions of Pa. respectively, and y is a non-negative parameter. In Eq. 10), the first term encourages both P and P to agree with each other, thus converging to the consistency. The second term enforces different dimensions of P and P to uniformly distribute over the latent space, thus avoiding the issue of model collapse. As a result, Eq. (10) is available to extract the consistent information between homophilous representations and heterogeneous representations.
In addition to extracting the consistent information, we aim to preserve the distinct characteristics of homophilous and heterogeneous representations well maintain their related information. However, we cannot directly add such regularization or constraints on both P and P. as its goal differs from that of Eq. (TO) and may lead to conflicts. To solve this issue, we propose to learn a projection head 9, to map both homophilous representations and heterogeneous representations
into another latent space, i.e., Q q, (Z) and Q = 9- (Z). After that, we employ transformation head us on Q to obtain Q' = us (Q). We further design specificity loss preserve the specific information related to homophilous representations and heterogeneous representations by:
N
N
Lape
(a' Σ
Σ
((ga. qn)² + (q'n -Pn)²).
(11)
where 17 is non- non-negative parameter. In Eq. the first term aims align Q with Q', instead of aligning Q with Q. As a result, it avoids directly aligning the projection of homophilous rep- resentations and heterogeneous representations to preserve their distinct information. Actually, if us is an identity transformation, the first term in Eq. (TT) is the same as the first term in Eq. (10). In addition, even if ug is not an identity transformation, Q' may also be equal to P, leading to the redundancy with the consistency loss. To avoid such scenarios, the second term in Eq. IT enforces the transformed Q' different from the original Q and P. As a result, Eq. (11) maintains the respective task-related information of homophilous representations and heterogeneous representations.
We integrate the consistency loss with the specificity loss to have the final objective function as:
J = Loom + XLspe;
(12)
where À is non-negative parameter. Finally, we concatenate homophilous representations Z with heterogencous representations Z to obtain final representations Ž for downstream tasks. As result, the concatenated representations Ž considering both homophily and heterogeneity in the heterogeneous graph can be theoretically proved to introduce more task-related information by Theorem 2.3] whose proof can be found in Appendix C.5
Theorem 2.3. For any downstream task T, the representations with both homophily and heterogeneity (e.g.. 2) contain more task-related information than the representations with only homophily (e.g., Z) or with only heterogeneity (e.g., Z), Le..
(2,T)) 2 where T(:.)) indicates the mutual information.
T)),
(13)

---

Published as a conference paper at ICLR 2024
Table Classification performance (i.e.,
Macro-FI and Micro-F1) on heterogeneous graph datasets.
ACM
Yelp
DBLP
Aminer
Method
Macro-F1
Micro-F1
Macro-F1
Micro-FI
Macro-FI
Micro-F1
Macro-FI
Micro-Fi F1
Deep Walk
73.9110.3
14.110.1
68.7.1..1
73.2.10.9
88.1110.2
89.5.10.3
54.7.10.8
59.7.10.7
GCN
86.9100.2
87.0.0.3
85.0.10.6
87.410.8
90.2.10.2
90.9110.5
64.5110.7
71.5110.9
GAT
85.0+1..4
84.9+0.3
86.4=0.5
88.2=0.77
91.0=#0.4
92.1+0.2
63.8+0.4
70.6
Mp2vec
87.6:10.5
88. 18.13
78.2:10.8
83.6:10.9
85.7:10.3
87.64:0.6
58.7::0.5
65.3110.6
HAN
89,4::0.2
89.2+-0.2
90.5+112
90.7=1.4
91.2++0.4
92.0+-0.5
65.3+40.7
72.8+10.4
HGT
91.5:1..7
91.610.6
89.9
90.2.10.6
90.9:1066
91.711..8
64.5+0..
71.0:0.7
DMGI
89.810.11
89.810.1
82.920.8
85.8=0.9
92.1+0.2
92.91103
63.840.4
67.640.55
DMGlattn
88.710.33
88.7
82.810.7
85.4
90.110.2
91.810..
62.4:0.9
66.810.8
HDMI
90.1++0.3
90.1+0.3
80.7+0.6
84.0=0.9
91.3++0.2
92.2+00.5
65.9+0.4
71.7++0.
HeCo
88.3+00.3
88.2+0.2
85.3+0.7
87.9=0.6
91.0++0.3
91.6+00.
71.8+0.9
78.6+0.7
HGCML
90.6:+0.7
90.7+0.5
90.7+0.8
91.0==077
91.9++0.8
93.2+0..7
70.5+0.4
76.3+0.6
CPIM
91.4.10.3
91.310.2
90.2.10.5
90.3.1.4
93.2.0.66
93.8.0.88
70.11009
75.811111
HGMAEE
90.5.10.5
90.6100.7
90.510.7
90.7.0.55
92.910.5
93,410.6
72.310.9
80,31122
DMG
91.0110.3
90.910.4
90.8.10.5
91.2-10.6
93.310.2
94.0110.3
72.010.7
79.510,9
HERO
92.2+10.5
92.1
92.4=0.7
92.3-0.6
93.8+0.6
94.410.4
75.1
84.5+0.9
Theorem 2.3 indicates that considering both homophily and heterogeneity introduces more task- related information than considering them individually, to benefit downstream tasks (verified by the Corollary in Appendix C.6p. Therefore, the proposed method is expected to perform better different downstream tasks than previous SHGL methods that consider only the homophily in the heterogeneous graph (verified in Section 3.2
EXPERIMENTS
In this section, we conduct experiments on both heterogeneous and homogeneous graph datasets to evaluate the proposed HERO in terms of different downstream tasks (i.e., node classification and similarity search), compared to heterogeneous and homogeneous graph methods. Detailed settings are shown Appendix D Additional experimental results are shown Appendix E
3.1 EXPERIMENTAL SETUP
3.1.1 DATASETS
The used datasets include five heterogeneous graph datasets and four homogeneous graph datasets. Heterogeneous graph datasets include three academic datasets (i.e., ACM (Wang et al., 20199) DBLP (Wang et al., 2019), and Aminer (Hu et al.] 2019)), one business dataset (i.e., Yelp (Lu et al. 2019h).
and one huge knowledge graph dataset (i.e., Freebase (Lv et al. (00TT). Homogeneous graph datasets include two sale datasets (iee,, Amazon-Photo and Amazon- Amazon-Computie Computers (Shchur et al. 2018) and two co-authorship datasets (i.e., Coauther-CS and Coauther-PPi Physics (Sinha et al. 2015)).
3.1.2 COMPARISON METHODS
The comparison methods include eleven heterogeneous graph methods and twelve homogeneous
graph methods. The former includes two supirvupe methods (i.e., HAN (Wang et al. 2019)
and HGT (Hu et al., (0200). one traditional unsupervised method (i.e.. Mp2vec (Dong et al. 201
and eight self self-supervised methods (i.e., DMGI (Park al.) 2020). DMGlatt (Park et al. [0220).
HDMI (Jing et al. (2021). HeCo (Wang et al.) 2021). HGCML (OMML(WWan (Wang et al.) 2023). CPIM (Mo et al.) 2023b), HGMAE (Tian et al.) (023), and DMG (Mo et al., 2023a)). The latter includes two semi- supervised methods (GCN (Kipf & Welling 2017) and GAT /Velickovic al. 2018)), one traditional unsupervised method (i.e., DeepWalk (Perozzi et al.] (014)). and nine self supf-supe methods, (i.e., DGI (Velickovic al. 2019). GMI (Peng et al. (222), MVGRL (Hassani & Khasahmadi, 2020). GRACE (Zhu et al. 2020b), GCA (Zhu et al. 1).1). GIC (Mavromatis & Karypis 2021). G-BT
(Bielak et al. 2022
COSTA (Zhang et al. 2022b), and DSSL. (Xiao et al. 2022)..
7

---

Published as a conference paper at ICLR 2024
Table Classification performance (i.e. Macro-Fl and Micro-F1) on homogeneous graph datasets. where OOM indicates Out-Of-Memory.
Amazon-Photo
Computrs
Coauther-CS
Coauther-Physich
Method
Macro-Fl
Micro-Fl
Macro-F1
Micro-FI
Macro-F1
Micro-F1
Macro-F1
Micro-FI
Deep Walk
87.4.10.5
89.7.0.33
84.0.10.3
85.6110.4
81.110.5
84.6.00.7
90.410.6
91.8.00.5
GCN
90.5.10.3
92.5.10.2
84.0.10.4
86.410..3
90.1100.8
93.0.10.5
93,810.6
95.6.10.6
GAT
90.2.10.55
91.8.10.4
83.2.1.22
85.7110.4
89.710.33
92.3.0.5
93.610.77
95.5.10.5
DGI
89.3:10.2
91.6.10.3
79.3
83.9100.5
89.410.66
92.2.10.8
93.2.10.7
94.5
GMI
89.3:10.44
90.6.10.2
80.110.4
82,2::0.4
OOM
0OM
0OM
OOM
MVGRL
90.1+0.3
91.7440.4
84.641..
86.91-0.5
89.3+.0.4
92.1
93.611..6
95.340.7
GRACE
90.3+0.5
91.911..3
84.2.10.3
86.810.5
90.2+0.8
93.0=0..6
94.2+0.5
95.7=0.4
GCA
91.1=0.44
92.410.44
85.910.5
87.710.3
90.1
92.910.5
94.120.6
95.7
GIC
90.0=0.33
91.6++.2
82.641.44
84.911.33
87.3+0.4
90.5#0.6
93.1+0.3
93,9
G-BT
91.5++0.4
92.6+0.6
86.2+0.3
88.1+0.5
90.0+0.7
93.0+0.5
93.0++.66
95.1+0.4
COSTA
91.3++0.4
92.5+0.3
86.4+0.3
88.3+0.4
90.1+0.6
93.0+0.7
94.0+0.4
95.7+0.6
DSSL
90.6+0.2
92.1+003
85.6+0.3
87.3+0.4
89.5+0.3
92.2+0.4
93.3+0.5
95.2+0.4
HERO
91.8.10.4
93.0.1..3
85.710.6
88.410.55
90.6
93.3.0.6
94.6.10.7
96.0
For fair comparison, we follow (Dong et al. 2017 Wang et al. 2019 Lu et al. 2019 Lv et al. 202T) to select meta- testt--pah-bbssee graphs for previous meta- meta-path-basee pash- SHGL methods. Moreover, we follow (Mo et al.) (023a) to conduct homogeneous graph methods on heterogeneous graph datasets by separately learning the representations of each meta-path- pata--aat-baaed graph and further concatenating them for downstream tasks. In addition, replace the heterogeneous encoder fe with GCN to implement the proposed method on homogeneous graph datasets because there is only one node type in the homogeneous graph. The code is released htthtss: http://///thttuttt/te com/ Yu fieeeieo//// MORM//ECCC HERC
3.2 RESULTS ANALYSIS
3.2.1
EFFECTIVENESS ON THE HETEROGENEOUS GRAPH
We first evaluate the effectiveness of the proposed method on the heterogeneous graph datasets by reporting the results of node classification (i.e., Macro-FI and Micro-FI) in Table and Appendix E and reporting the results of similarity search (i.e., Sim@5 and Sim @ 10) in Appendix E Obviously, our method achieves superior performance on both node classification and similarity search tasks.
First, for the node classification task, the proposed method always outperforms the comparison meth- ods by large margins. For example, the proposed method on average, improves by 2. 1,, compared to the best SHGL method (i.e., DMG), four heterogeneous graph datasets. The reason can be attributed to the fact that the proposed method extracts both homophily and heterogeneity in the het- erogeneous graph, thus introducing more task-related information to improve the effectivvness the classification task. Second, for the similarity search task, the proposed method also obtains promising improvements. For example, the proposed method on average, improves by 1. 1.8%, compared the best SHGL method (i.e., DMG), on four heterogeneous graph datasets. This demonstrates the superi- ority of the proposed method, which captures the homophily the heterogeneous graph, enforcing the representations to have the grouping effect and thus increasing the similarity of nodes within the same class. As a result, the effectiveness of the proposed method is verified on the heterogeneous graph datasets in terms of different downstream tasks.
3.2.2 EFFECTIVENESS ON THE HOMOGENEOUS GRAPH
We further evaluate the effectiveness of the proposed method on the homogeneous graph datasets by reporting the results of node classification (i.e., Macro-FI and Micro-Fl) in Table 2 We can observe that the proposed method achieves competitive results the homogeneous graph datasets.
First, compared to the semi-superrised baselines (i.e., GCN and GAT), the proposed method always achieves the best results. For example, the proposed method on average, improves by 1.1%, compared to the best cemi-supervised method (i.e.. GCN), on four homogeneous graph datasets. Second, compared to the self-supervised methods, the proposed method also achieves superior performance.
8

---

Published as a conference paper at ICLR 2024
(a) ACM
(b) Yelp
(c) Example graph of ACM
(d) Example graph of Yelp
Figure 3: (a) and (b) indicate node correlation maps of ACM and Yelp datasets reordered by node labels. (c) and (d) indicate node correlation maps and corresponding visualizations (top 30% values in the correlation map are visualized as edges) of example graphs of ACM and Yelp datasets.
For example, the proposed method outperforms the best self-supervised method (i.e., COSTA), on almost all homogeneous graph datasets. This indicates that the proposed method extracts the consistent information between the original graph and the graph with homophily, well preserves the specific information within each them to benefit downstream tasks. As a result, the effectiveness of the proposed method is further verified the homogeneous graph datasets.
3.2.3 VISUALIZATION AND CASE STUDY
To verify the grouping effect of homophilous representations, visualize the node correlation maps of ACM and Yelp datasets in Figures 3(a] and 33b] where rows and columns are reordered by node labels. In the correlation map, the darker a pixel, the higher the correlation between nodes. In Figures 3 and 3(b] the correlation maps exhibit block diagonal structure where the nodes of each block belong to the same class. This indicates that if two nodes belong to the same class, then the correlation of their representations will be high, i.e., their representations are expected to be aligned. This verifies the grouping effect of the homophilous representations, which capture the homophily in the heterogeneous graph.
Visualization of Grouping Effect.
Visualization of Example Graph. To further verify that the homophilous representations indeed capture the homophily the heterogeneous graph, we sample example graphs from ACM and Yelp datasets and visualize the correlation maps among sampled nodes in Figures 3(c) and 3(d) Moreover, we visualize the top 30% values in the correlation maps as edges between nodes to make them more intuitive. In Figures [(c and 3(d have the observations follows. First, two nodes (e.g.. node and node 11) within the same class do have high correlation values, while two nodes (e.g., node and node 1) from different classes do have low correlation values. Second, the visualized edges in Figures and ((d) indeed show a high homophily rate (e.g., 73% in the example graph of the Yelp dataset). This further verifies that the homophilous representations indeed capture the homophily in the heterogeneous graph without pre-defined meta-paths.
4 CONCLUSION
In this paper, we proposed self-supervised heterogeneous graph learning framework to capture both homophily and heterogeneity in the heterogeneous graph without pre-defined meta-paths. To do this, we proposed to learn a self-expressive matrix adaptively and employ the heterogeneous encoder to obtain homophilous and heterogeneous representations for capturing homophily and heterogeneity in the heterogencous graph, respectively. We further designed the consistency loss and the specificity loss to extract the consistent information between homophilous representations and heterogeneous representations and to maintain their specific information in different latent spaces, respectively. Theoretical analysis indicates that the homophilous representations capture the homophily in the heterogeneous graph. In addition, the fused representations are provable to contain more task-related information than the representations with homophily or heterogeneity only, thus benefiting downstream tasks. Extensive experimental results demonstrate the effectiveness of the proposed method on both homogeneous and heterogeneous graph datasets in terms of different downstream tasks. We discuss potential limitations and future work in Appendix F

---

Published as a conference paper at ICLR 2024
5 ACKNOWLEDGMENTS
This work was supported in part by the National Key Research and Development Program of China under Grant 2022YFA 1004100, in part by the Deedic--nngineering Cooperation Funds from University of Electronic Science and Technology of China under Grant ZYGX2022GRHOMK and Grant (GGX2022YGRHOV in part by the National Natural Science Foundation of China under Grant 62276052.
REFERENCES
Piotr Bielak, Tomasz Kajdanowicz, and Nitesh V. Chawla. Graph barlow twins: A self-supervised representation learning framework for graphs. Bhowledge-Baseed Systems, 256:10963 2022.
Olivier Chapelle, Jason Weston, and Bernhard Schölkopf. Cluster kernels for semi-supervised learning. NeurIPS, volume 15, 2002.
Bo Chen, Jing Zhang, Xiaokang Zhang, Yuxiao Dong, Jian Song, Peng Zhang, Kaibo Xu, Evgeny Kharlamov, and Jie Tang. Gcead: Graph contrastive coding for anomaly detection. IEEE Transac- lions Knowledge and Data Engineering, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pp. 1597-1607, 2020.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, et al. Masked language modeling for proteins via linearly scalable long-context transformers. In ICLR, 2020.
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljacic. Equivariant contrastive learning. In ICLR, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pp. 4171-4186, 2019.
Yuxiao Dong, Nitesh V. Chawla, and Ananthram Swami. metapath2vec: Scalable representation learning for heterogeneous networks. In SIGKDD, pp. 135-144, 2017.
Meir Feder and Neri Merhav. Relations between entropy and error probability. IEEE Transactions Information Theory, (0(1):259- 266, 1994.
Andrea Galassi, Marco Lippi, and Paolo Torroni. Attention in natural language processing. IEEE Transactions on Neural Networks and Learning Systems, (2(10):491-4300) 2020.
Sean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurlPS, volume 33, pp. 21271 11211-22286 21244, 2020.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In ICML, pp. 4116-4126, 2020.
Xiaofei He, Shuicheng Yan, Yuxiao Hu, and Hong-Jiang Zhang. Learning a locality preserving subspace for visual recognition. In ICCV, pp. 385-392, 2003.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilddo Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. I Learning deep representations by mutual information estimation and maximization. In ICLR, pp. 1-24, 2019.
Binbin Hu, Yuan Fang, and Chuan Shi. Adversarial learning on heterogeneous information networks. In SIGKDD, pp. 120-129, 2019.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In www, pp. 2704 2710, 2020.
10

---

Published as a conference paper at ICLR 2024
Di Jin, Luzhi Wang, Yizhen Zheng, Guojie Song, Fei Jiang, Xiang Li, Wei Lin, and Shirui Pan. Dual intent enhanced graph neural network for session-based new item recommendation. In WWW, pp. 684-693, 2023.
Baoyu Jing, Chanyoung Park, and Hanghang Tong. Hdmi: High-order deep multiplex infomax. In WwW, pp. 2424, 2021.
Diederik Kingma and Lei Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
N. Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, pp. 1-14, 2017.
Tassilo Klein and Moin Nabi. Contrastive self-supervised learning for commonsense reasoning. arXiv preprint arXiv:2005.0066 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations, In ICLR, pp. 1-14, 2020.
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with back- propaproongttion network. In NeurIPS. volume 2, 1989.
Xiang Li, Ben Kao, Caihua Shan, Dawei Yin, and Martin Ester. Cast: a correlation-based adaptive spectral clustering algorithm multi-scale data. In SIGKDD, pp.
2020.
Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, Xinwang Liu, and Fuchun Sun. A survey of knowledge graph reasoning on graph types: Static, dynamic, and multimodal. arXiv preprint arXiv:2212.0576: 2022.
Ke Liang, Sihang Zhou, Yue Liu, Lingyuan Meng. Meng Liu, and Xinwang Liu. Structure guided multi-modal pre-trained transformer for knowledge graph reasoning. arXiv preprint vrxiv:2307.03591, 2023.
Ke Liang. Yue Liu, Sihang Zhou, Wenxuan Tu, Yi Wen, Xihong Yang. Xiangjun Dong, and Xinwang Liu. Knowledge graph contrastive learning based on relation-symmettee Transactions Knowledge and Data Engineering, 36(1 (:(1)):226220)) 238, 2024.
structure. IEEE
Meng Liu, Ke Liang, Yawei Zhao, Wenxuan Tu, Sihang Zhou, Xinwang Liu, and Kunlun He. Self- supervised temporal graph learning with temporal and structural intensity alignment. arXiv preprint arXiv:2302.0749) 2023a.
Meng Liu, Yue Liu, Ke Liang, Siwei Wang. Sihang Zhou, and Xinwang Liu. Deep temporal graph clustering. arXiv preprint arXiv:2305. 10738, 2023b.
Yuanfu Lu, Chuan Shi, Linmei Hu, and Zhiyuan Liu. Relation structure-aware heterogencous information network embedding. In AAAI, pp. 4456 4463, 2019.
Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng, Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, and Jie Tang. Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks. In SIGKDD, pp. 1 1150-1160, 2021.
Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily necessity for graph neural networks? In ICLR, 2021.
Costas Mavromatis and G. Karypis. Graph infoclust: Maximizing coarse-grain mutual information in graphs. In PAKDD, pp. 541-553, 2021.
Yujie Mo. Yajie Lei, Jialie Shen, Xiaoshuang Shi, Heng Tao Shen, and Xiaofeng Zhu. Disentangled multiplex graph representation learning. In ICML, volume 202, pp. 24983-25000 2023a.
Yujie Mo, Zongqian Wu, Yuhuan Chen, Xiaoshuang Shi, Heng Tao Shen, and Xiaofeng Zhu. Multiplex graph representation learning via common and private information mining. In AAAI, pp. 9217-9225, 2023b.
11

---

Published as a conference paper at ICLR 2024
Naila Murray and Florent Perronnin. Generalized max pooling. In CVPR, pp. 2473 2473-2480, 2014.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03745. 2018.
Erlin Pan and Zhao Kang. Multi-view contrastive graph clustering. In NewrIPS, volume 34, pp. 2148-2155 2159, 2021.
Chanyoung Park, Donghyun Kim, Jiawei Han, and Hwanjo Yu. Unsupervised attributed multiplex network embedding. AAAI, pp- 3371,5378, 2020.
Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In www, pp. 259-270, 2020.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representa- tions. In SIGKDD, pp. 701-710, 2014.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günneemann Pitfalls of graph neural network evaluation. arXiv preprint arXiv: 1.05668, 2018.
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June (Paul) Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In WWW, pp. 243-246, 2015.
Felix Stahlberg. Neural machine translation: A review. Journal Artificial Intelligence Research, 69:343 418, 2020.
Xiangguo Sun, Hongzhi Yin, Bo Liu, Hongxu Chen, Jiuxin Cao, Yingxia Shao, and Nguyen Quoe Viet Hung. Heterogeneous hypergraph embedding for graph classification. In WSDM, pp. 725-733, 2021.
Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan, All in one: Multi-task prompting for graph neural networks. In SIGKDD, 2023a.
Xiangguo Sun, Hong Cheng, Bo Liu, Jia Li, Hongyang Chen, Guandong Xu, and Hongzhi Yin. Self-supervised hypergraph representation learning for sociological analysis. IEEE Transactions on Knowledge and Data Engineering, 2023b.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh Chawla. Learning mips on graphs: A unified view of effectiveness, robustness, and efficiency. In ICLR, 2022.
Yijun Tian, Kaiwen Dong, Chunhui Zhang, Chuxu Zhang, and Nitesh V. Chawla. Heterogeneous graph masked autoencoders. In AAAI, pp. 9997-10005, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998-6008, 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, pp. 1-12, 2018.
Petar Velickovie, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In ICLR, pp. 1-17, 2019.
Ehsan Elhamifar René Vidal. Sparse subspace clustering. In CVPR, volume 6, pp. 2797, 2009.
Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint trXiv:2006.04766 2020.
Xiao Wang, Houye Ji. Chuan Shi. Bai Wang. Yanfang Ye, Peng Cui, and Philip S. Yu. Heterogeneous graph attention network. In WWW, pp. 2022-2000 2032, 2019.
Xiao Wang, Nian Liu, Hui Han, and Chuan Shi. Self-supervised heterogeneous graph neural network with co-contrastive learning. In SIGKDD, pp. 1726-1736, 2021.
12

---

Published as a conference paper at ICLR 2024
Zehong Wang. Qi Li, Donghua Yu, Xiaolong Han, Xiao-Zhi Gao. and Shigen Shen. Heterogeneous graph contrastive multi-view learning. In SDM, pp. 336-144,22 2023.
Max A Woodbury. Inverling modified matrices. Department of Statistics, Princeton University, 1950.
Teng Xiao, Zhengyu Chen, Zhimeng Guo, Zeyang Zhuang, and Suhang Wang. Decoupled self- supervised learning for graphs. In NeurlPS, 2022.
Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Xihong Yang. Xiaochang Hu, Sihang Zhou, Xinwang Liu, and En Zhu. Interpolation-based contrastive learning for few-label semi-supervised learning. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-12, 2022. doi: 10. STW.:NNNNN..222..111188
Xihong Yang. Yue Liu, Sihang Zhou, Siwei Wang, Wenxuan Tu, Qun Zheng, Xinwang Liu, Liming Fang, and En Zhu. Cluster-guided contrastive graph clustering network. In AAAI, volume 37, pp. 10834-10842, 2023.
Dingjun Yu, Hanli Wang, Peiqiu Chen, and Zhihua Wei. Mixed pooling for convolutional neural networks. In RSKT, pp. 364-375, 2014.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, volume 139, pp. 12310-12320, 2021.
J. Zhang, L. Gao, X. Luo, H. Shen, and J. Song, Deta: Denoised task adaptation for few-shot learning. In ICCV, pp. 11507-11517, 2023a.
Zhang, Lianli Gao, Bingguang Hao, Hao Huang, Jingkuan Song, and Hengtao Shen. From global to local: Multi-scale out-of-distribution detection. IEEE Transuctions on Image Processing, 32: 6128, 2023b.
Rui Zhang, Arthur Zimek, and Peter Schneider-Kante Kamp. A simple meta-path-free framework for heterogeneous network embedding. In CIKM, pp. 2600-2609, 2022a.
Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. Costa: Covariance-preservii feature augmentation for graph contrastive learning. In SIGKDD, pp. 2524-2534, 2022b.
Jianan Zhao, Xiao Wang. Chuan Shi, Binbin Hu, Guojie Song, and Yanfang Ye. Heterogeneous graph structure learning for graph neural networks. In AAAI, pp. 4697-4705, 2021a.
Jiawei Zhao, Ke Yan, Yifan Zhao, Xiaowei Guo, Feiyue Huang, and Jia Li. Transformer-base dual relation graph for multi-label image recognition. In CVPR, pp. 163-172, 2021b.
Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. NeurIPS, volume 16, 2003.
Sheng Zhou, Kang Yu, Defang Chen, Bolang Li, Yan Feng, and Chun Chen. Collaborative knowledge distillation for heterogeneous information network embedding. In WWW, pp. 1631-1639, 2022.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In NeurIPS, volume 33, pp. 7793-7804, 2020a.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint 2001::2006.04131, 2020b.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. WWW, pp. 2069-2080, 2021.
13

---

Published as a conference paper at ICLR 2024
A
RELATED WORK
This section briefly reviews topics related to this work, including self- self-supervised learning in Section heterogeneous graph learning in Section A.2. and self-attention mechanism in Section A.3
A.1 SELF-SUPERVISED LEARNING
Self-supervieed learning (SSL) has emerged as a promising approach to address the challenge of acquiring labeled data for training deep neural networks. Unlike traditional supervised learning, where labeled data required, SSL leverages the abundance of unlabeled data to learn meaningful representations. Therefore, SSL has shown strength in various domains such as computer vision
(Grill et al. 2020 Yang et al. 2022 Zhang et al.) 2023a), natural language processing (Devlin et al. 2019: Klein & Nabi, 2020; Lan et al. 2020), and graph representation learning (Jin et al. 2023
et al.
2023b Sun et al., 2023a Liang et al. 2024).
Previous SSL methods generally conduct contrastive learning between the positive pairs and neg- ative pairs to maximize the mutual information between the local representations and their related representations. For example, Deep InfoMax (Hjelm et al.) 2019) conducts contrastive learning by maximizing the mutual information between a local patch and its global context. CPC (Oord et al.) 2018) achieves great results on the speech recognition task by maximizing the mutual information between the local audio representations and its global audio representations. SimCLR (Chen et al. 2020) conducts the contrastive learning between the original view and the augmented view, and argues that data augmentation plays an important role in contrastive learning. (Sun et al.) 2023b) proposes a novel hypergraph neural network learn influence flowing under social criteria with dual contrastive loss. To remove the need for negative samples in contrastive SSL methods, recent works propose to conduct sulf-supprvised learning without negative samples. For instance, BYL (Grill al., 2020) leverages target network to generate target representations for augmented view, and the online network aims to match its own predictions with the target representations. Barlow Twins (Zbontar et al.) (021) propose to capture the invariant information between the original view and the augmented view as well as decorrelate different representation dimensions. E-SSL (Dangovski et al. 2022) encourages the equivariance to some transformations, while maintaining the invariance to other transformations, thus improving the semantic quality of representations.
A.2 HETEROGENEOUS GRAPH LEARNING
Heterogeneous graph learning methods aim to explore the latent patterns in the heterogeneous graph and have been applied to various practical applications. Considering the cost of obtaining node labels. self-supervised heterogeneous graph learning has drawn much attention (Liang et al., 2022] Tian et al. 2022 Liu et al. al. [202aa] Yang et al. [023 Wang et al. 2023). Existing SHGL methods generally employ meta-paths to extract distinct semantic relationships between nodes of the same type while ignoring other node types.
In light of such meta-pan -ath preprocessing, existing SHGL methods can be broadly categorized into two groups, i.e., intra-pall tat-pathhhh learning methods and inter-pat patth learning methods. Intra-path learning methods focus on capturing the global properties within each meta-path--ae based graph to enhance the quality of node representations. For instance, DMGI (Park et al. 2020) and HDMI (Jing al. 2021) utilize contrastive learning techniques to establish connections between node representations and graph summaries, compelling the incorporation of global properties within the node representations. Inter-path learning methods aim to capture the associations as well as invariant information across different meta-path pett--pth-bbsed based graphs. For example, HGCML (Wang et al., 2023) and CPIM (Mo et al.) 2023b) propose to maximize the mutual information between node representations from different meta-path-basee graphs. Although existing SHGL. methods have shown great potential and achieved impressive results in various tasks, they generally require pre-defined meta- meta-paths, which induce substantial expert knowledge and expensive costs. Moreover, many previous SHGL methods ignore the heterogeneity in the heterogeneous graph to lose task-related information. Even though a few works (Wang et al., (022)) explore the neighbors of nodes from different types in the network schema, they directly enforce the homophily of meta-paths and the heterogeneity of the network schema to align with each other by contrastive learning. As a result, they still need pre-defined meta- meta-paths and may lose distinct information in the homophily and heterogeneity to weaken downstream tasks.
14

---

Published as a conference paper at ICLR 2024
A.3 SELF-ATTENTION MECHANISM
Self-attention is a mechanism that allows a model to focus on all samples within a sequence or set and learn relationships between them. It enables the model assign varying levels of importance or attention to different samples based on their relevance to each other (Velickovic et al. 2018 Devlin al., 2019 Liang al., 2023 Zhang et al. 2023b). As a result, the self-attention mechanism has achieved promising results in natural language processing tasks (e.g., text classification (Galassi ct al.
2020), and machine translation (Stahlberg 2020)) and computer vision tasks (e.g, image recognition
and object detection (Zhao et al. 2001bb).
Actually, the sellf-atenntinn mechanism closely related the notion of the self eelf-expressive matrix in the proposed method, wherein each sample is expressed as linear combinations of all other samples, thus capturing the global correlation among samples. However, there are also some major differences between them. First, the self-expressive matrix is not restricted to be non-negen non-negative, allowing for both positive and negative attention, while the self-attention mechanism is restricted to be non-negative. Second, the self-attention mechanism is defined as function of the tokens that are parameterized by learnable weights. Instead, the self-expressive matrix in the proposed method is the closed-form solution derived from unsupervised objective function without parameters. Third, the self-expressive matrix in the proposed method is regularized by the feature distance matrix and is encouraged focus on the most relevant neighbors of each sample instead of arbitrary neighbors in the self- attention mechanism. Fourth, the time complexity of the closed-form solution of the self-expressive matrix can be reduced to scale linearly with the sample size by the matrix identity transformation, while the self-attention mechanism requires quadratic time complexity with the sample size. Even though part of works (Wang et al. 2020, Choromanski et al. 2020) propose to reduce the time complexity of the self-attention mechanism, they sacrifice the model performance by the approximation transformation. As a result, the self-expressive matrix is more general and flexible than the the heterogeneous graph.
mechanism, and therefore we employ it to capture the homophily in
B ALGORITHM AND COMPLEXITY ANALYSIS
This section provides the pseudo of the proposed method in Section B.I and the complexity analysis of our method in Section B.2
B.1 ALGORITHM
Algorithm 1 The pseudo-code of the proposed method.
Input: Heterogeneous graph G EV, T, T,R), non-negative parameters a, 3, y, 7), and A;
Output: Encoders Is. fe; Initialize parameters;
2: while not converge do
3: Obtain node representations with encoder 9ø:
4: Obtain the closed-form solution of self-expressive matrix S" by Eq. (4)
5: Obtain homophilous representations Z by Eq. (5):
6: Transform the matrix and reorder the matrix multiplication to reduce the time complexity by
Eq. O
7:
Obtain heterogeneous representations with encoder fo;
8: Project homophilous representations and heterogeneous representations into latent space to obtain P and P:
9:
Conduct the consistency loss between P and P by Eq. 10:
Project homophilous representations and heterogeneous representations into another latent space to obtain Q and Q:
11:
Transform Q with transformation head to obtain Q':
12:
Conduct the specificity loss between Q' and Q by Eq.
13:
Compute the objective function J by Eq. (12;;
14:
Back-propagate J to update model weights;
15:
end while
15

---

Published as a conference paper at ICLR 2024
B.2 COMPLEXITY ANALYSIS
Based on the Algorithm| 1 Jbove, we then analyze the time complexity of the proposed method. We define and d as the number nodes, and dimension of representations, respectively. According to Eq. 66) we can obtain the homophilous representations Z as follows.
Z H H (
"B a DB,
(14)
where H(+1), B E NNx
B =
H
HU
and La E RRdxd indicates the identity matrix. Therefore, we first calculate B and calculate the second term of it from right to left. Specifically, the matrix inversion is conducted on matrix iRdxd whose time complexity is (dd)) Therefore, the overall time complexity of calculating B is O(Nd + + After that, we obtain the homophilous representations with Eq. (14). In Eq. (14p. the time complexity to calculate DB is O(kd), where k indicates the nonzero entries of the sparse feature distance matrix D and < N In addition, the time complexity of calculating H (H (H is ((Nd') Therefore, the time complexity of calculating homophilous representations Z is ONd + d + kd). Moreover, the time complexity of Eq. I0] is ((,)), and the time complexity of Eq. 111 is ((N). Therefore, the overall complexity of our algorithm is ONNd + d + kd) in each epoch, which is scaled linearly with the sample size.
(E
C DERIVATION PROCESS AND PROOFS OF THEOREMS
This section provides definition, detailed derivation process, and proofs of Theorems in Section 2] including the definition of homophily mining. homophily ratio, and heterogeneity mining in Section the derivation process of the closed-form solution in Section C.2. matrix transformation in
C.I.
Section
C.3 proof of Theorem 2.2 in Section C.4] proof of Theorem 2.3 in Section C.5) and the
Corollary as well as its corresponding proof in Section C.6
C.I DEFINITIONS
According to previous works (Zhu et al. 2020a Ma et al. 2021), we have the definition follows.
Definition C.1. (Homophily mining) Given a graph G (V,E), the homophily mining is the comnectivity utilization and information aggregation among nodes within the same class. Formally, we have
m (1) (ES (In (1s, E (.
(15)
where indicates the 1-th layer, MES indicates the message passing function, m, and h, indicate node representations of node vi and node vy Yt and yj indicate labels of node v; and node Gj.
Definition C.2. (Homophily ratio Given graph G = [V, J. the edge homophily ratio defined as the fraction of edges that connect nodes within the same class. Formally, have
h
Σ (N..P))EE
1 v))
(16)
where [8] is the number of edges in the graph and (-) is the indicator function.
Definition C.3. (Heterogeneity mining) Given a heterogeneous graph G = VV,EXXTT.7 the heterogeneity mining is the connectivity utilization and information aggregation among nodes among nodes from different types. Formally, we have
m MES(O) ((nf-") (1-1) (V), Er,o, E Vr.u, eV.
(17)
where indicates the 1-th layer, MES indicates the message passing function, m, and h, indicate node representations of node v; and node Uy E, indicates the r-th type of edges, V, and Vz. indicate the 7-th and r'tth type of nodes, and T + T'.
16

---

Published as a conference paper at ICLR 2024
C.2
CLOSED-FORM SOLUTION
Given the objective function in Eq.
3 we let
N
N
||H(+1) SH
2
Σ
xx
x,1β2115
s/=1
(18)
=Tr((H
)
aTTr(S ( sT(Sss)
where Tr(-)) indicates the trace of matrix. Then we have
aJ ( H 2SH 2
(H++))y + o. + 2.5.
(19)
as
Let Eq.
19
equal to 0, we can obtain the closed-form solution S of Eq. 3 i...
S" = = (H
S (H (1+1)
( + BIN)
(20)
C.3 MATRIX TRANSFORMATION
Given four matrix, i.e,, A E R E Baxk Ce E R and V E R**,, where n, are dimensions of these matrix, according to the Woodbury identity matrix transformation (Woodbury 1950), we have
(A +
A - VA- VA- -
(21)
Without loss of generality, the matrix A and C can be replaced with the identity matrix, therefore, we further have
UV)
U( VU)-IV.
(22)
Based on Eq. 22). we can transform (H++))
(11)) + ;IN) -1 in Eq. as:
(1+1) (H [N) 1
H
(La
1 ( -
(H(4)) (23)
Ba
Therefore, with Eq. and Eq.
23 we can transform Eq. as
Z = Z (++)
(1+) H u D)
H 1- + (IN) (++)
= (H (H*U)T LD)
H H(+ ( H( (H*+)) (((+)
((H+))
(24)
Then we replace ( 1/2 H(+ (I. + (H4 ( H(++1 with B and obtain
z = H ( 1
D)B
(25)
( U -BB.
This is exactly the Eq. (6)
C.4 PROOF OF THEOREM 2.2
Theorem C.4. (Restating Theorem 2.2 in the main text). Both self-expressive matrix N
and homophilous representations Z E lk xd huve the grouping effect for any two nodes v, and by
that hold the condition 2 + Uj. i.e.,
22 u,
1 spp 0, and Fig 2jal
0, 595d,1 Si.jss N.
(26)
17

---

Published as a conference paper at ICLR 2024
Proof. For S i I N, we denote s; the i-th row of the coefficient matrix S, and denote
d as the i-th row of the feature distance matrix D. According to the Eq. ( we let J(s)) =
[| (1+1)
(s. + - With the closed-form solution in Eq. (4). we have s [++D 1 (
8. = 0, v1 p N. Therefore, we have - (h. (4+1) s HH + + - dip + +
We further have
(h((t)) - HH ( 7 dd
(27)
B
Based on Eq.
27
we can obtain
(1+1)
hy (hg 1/ - (s)
(1+1)
87) H (11) (hp (h. (dipp dzp)
Spp
(28)
jp
3
Moreover, according to Eq. we have
S = (b.
H 11))
a
( + (NN
(29)
Therefore,
S5 sj ((h - by (
a (d: (H)) (1(+)) + Il)
(30)
2
Replace (H (++1) (H(+)((H(++))))) + (IN) (++)) (hg / with M, then we can rewrite Eq. 28 as:
(h l1+1] - h (111) (bg 1
(hh h (+
( (+1))r (d. d
ip ip
B
(dip djp))
3
(h. (1+1) - h1 11+1)
((b,
(1))rr (H(+1)²M)
(d, d,jM
2 2d ddp)
3
B
3
(31)
Therefore, we have
[(h(--1) h. b((+1)))
||2 [1 ((b
9 Il(,
d
M|22
1
1ppl S
3
3
(32)
2/ ldip dipl
3
If the condition that es + uy holds, i.e. ||x; -xyl12 + hold, then we have Idip dipl + 0, 11 < d + O. Moreover, we can obtain [[ (2+1) 1(+1))l2 + 0 according to Eq. O.. As a result, the R.H.S. of Eq. (32) close to 0, and 1s - sppl - 0 holds, thus S has the q d. denote h as the q-th column of (!-1), we have Zig Zjg = 8²h9 - h = (s - sj)h.. Therefore, we further have |zig 2jgl S ||s -s]l2| 2- If the condition that 8; + Uj holds, we obtain || - 87 12 - 0 according to Eq. (30p. That is, W + by holds, then |zig Zjq + 0 holds with V1 < S d, thus homophilous representations Z have the grouping effect. Therefore, we complete the proof.
p S N, ie.. ||d, d
grouping effect. In addition, for any
C.5 PROOF OF THEOREM 2.3
In the following proofs, for random variables A, B, C, we use I(A, B) to represent the mutual information between A and and we use I (AA to represent conditional mutual information of A and B on given C, use (AA) for the entropy, and H (A(B) for the conditional entropy. We first list some properties of mutual information and entropy that will be used in the proofs.
* Property 1. Relationship between the mutual information and entropy:
T(A, B) H (AA ((A) B).
(33)
18

---

Published as a conference paper at ICLR 2024
Property 2. Relationship between the conditional mutual information and entropy:
[AA B C) H(A) C) - (( B,C).
(34)
Property 3. Non-negativity of mutual information:
(A, B) 2 0, (A, B C) 2.
(35)
Property 4. Relationship between the conditional entropy and entropy:
H(A B) (A, B) H(B).
(36)
Theorem C.5. (Restating Theorem 2.3 in the main text). For any downstream task T, the representa- tions with both homophily and heterogeneity (e.g.. Z) contain
task- information than the representations with only homophily (e.g., Z) or with only heterogeneity (e.g.. Z), i.e.
I(Z,T) 2 mZ, ((,
(37)
where -(: indicates the mutual information.
Proof. Given the fused representations that contain both homophily and heterogeneity, we have
=(²)= + ((Z))
(38)
where HZZ and H HZ indicate the specific information of Z and Ž, respectively, and I(Ž, Z) indicates the consistent information between Z and Z. According to Properties and 4, we have
(2) (ZZ + (2 [(Z Ž)
H(Z + (Z)Z - (ZZ) (1
(39)
(Z)Z)) + Hī H(Z, 2) (Z (Z
= ().
Therefore, for any downstream task T, we further have
1(2,T) (Z,,
(40)
To prove I(2, T) 2 max(T(Z, T), ((2, we first prove [(2, 2 T(Z,T). Then based on Eq.
39.. Eq. (40). Property 1, and Property 4. we can transform [2, as follows.
1(2,T) T) H(2) H([[T)
= -((2) - (2, + H(T)
(41)
HHZ,Z) H2Z, Z. H(T).
Moreover, based on Properties and 2, we have
(Z, = HZ) HZ
(42)
- H (Ž
(43)
=H(ZZ H()) - H (Z),T))
Then with Eq.
42
Eq.
and Property 4, we can obtain
I(Z,T)+ + 1 (ZZ) (())) + (Z, 2) (()) - (Z
.H Z) (((2)))
=H(Z, Z)
H(
(44)
=H(Z, Z) H(Z,T) + +H(T) H(Ż, Z, Z,T) (Z,
- =1 2) H(T) H(Ž,
19

---

Published as a conference paper at ICLR 2024
According to Eq. (1 and Eq. (44) have
1(2,T) I(Z,T) I(2, (Z,TZ) Z).
(45)
Based on Property 3, we have [Z, 2 0, the we can get
I(2,T) 2 (Z,T).
(46)
Similarly, we can also obtain
I(2,T) 2 I)
(47)
Therefore, 1(2, T) 2 max (() and we complete the proof.
L
C.6 COROLLARY C.6
To further verify Theorem 2.3 we take the classification an example downstream task T, then employ the Bayes error rate (Feder & Merhav 1994), which is the lowest achievable error when learning arbitrary classifier from the representation infer the labels. Specifically, let Pe be the Bayes error rate of arbitrary learned representations Ž and T as the prediction for T from the classifier. Thus, we have Pe Ez- [1 maxter P(T = 2) To prevent overflow, we further define P. Th(),), where Th(x) min/max(z, 0), 1/[T|} is threshold function. Then have the following Corollary:
Corollary C.6. For a classification task T, the representations with both homophily and heterogeneity (e.g.. 2) achieve a smaller supremum of Bayes error rate than the representations with only homophily (e.g., Z) or with only heterogeneity (e.g.. Z), i.e.,
(up(P.(2) < min(sup(P.(Z)).suup (Z))).
(48)
Proof. According to the inequality between (2( and H in previous work (Feder & Merhav 1994), we first have
log(1 - Pc()) (T 2).
(49)
Based on Property and Eq.
we have
H(T 2) H(T) I(T, 2)
(50)
= HTT) I(Z,T) -
Similarly, we further have
H(T Z) = - I(T,Z)
(51)
Based Property 3, have H(T) ((Z -
TI) H(T) (), and thus indicates
that the R.H.S. of Eq. (50) is smaller than the R.H.S. of Eq. (T). Therefore, the fused represen- tations Ž achieve smaller supremum of Bayes eιror rate than Z, i.e. sup(Pa(Z). S sup(P. (Z)). also obtain sup( (ž)) < sup(P:(Z)) Therefore, have sup(P:(Z)) S Similarly, sup( (Pa (Z))). and thus complete the proof. min(s (Pa (Z)),
L
D EXPERIMENTAL SETTINGS
This section provides detailed experimental settings in Section 3 including the description of all datasets in Section D.I. summarization of all comparison methods Section D.2 and model architectures and settings in Section D.3
20

---

Published as a conference paper at ICLR 2024
Table Statistics of all datasets.
Datasets
Type
#Nodes
#Node Types
#Edges
#Edge Types Target Node #Training
#Test
ACM
Heter
8,994
3
25,922
Paper
600
2.125
Yelp
Heter
3,913
4
72,132
Bussiness
300
2,014
DBLP
Heter
18,405
3
67,946
Author
800
2,857
Aminer
Heter
55,783
3
153,676
Paper
1,000
Freebase
Heter 180,098
8
1,645,725
62
Book
1,909
5,568
Amazon-Photo
Homo
7,650
1
238,162
Photo
765
6,120
Amazon-Coue Compute-s Homo
13,752
1
491,722
Computer
1,375
11,002
Coauthor-CS
Homo 18,333
1
163,728
Author
1,833
14,667
Coauthor- Coauthor-Physics Homo 34,493
495,924
Author
3,449
27,595
D.1 DATASETS
We use five public heterogeneous graph datasets and four public homogeneous graph datasets from various domains, Heterogeneous graph datasets include three academic datasets (i.e., ACM (Wang
al. 2019). DBLP Wang et al. 2019) and Aminer (Hu al. 2019). ). one business dataset (i.e. Yelp (Zhao et al. 202 and one huge knowledge graph dataset (i.e., Freebase (Lv et al., 2021
Homogeneous graph datasets include two sale datasets (i.e., Amazon- Photo and Jmazzon-Compur
(Shchur et al. 2018)), and two co-authorship datasets (i.e., Coauther-CS and Coauther-Phyyse (Sinha
al. 2015)). Table summarizes the data statistics. We list the details of the datasets as follows.
ACM an academic heterogeneous graph dataset. It contains three types of nodes (paper (P), author (A), subject (S)), four types of edges (PA, AP, PS, SP), and categories of papers as labels.
Yelp is business heterogeneous graph dataset. It contains three types of nodes (business (B), user (U), service (S), level (L)), six types of edges (BU, UB, BS, SB, BI. LB), and categories of businesses as labels.
DBLP an academic heterogeneous graph dataset. contains three types of nodes (paper (P), authors (A), conference (C)), four types of edges (PA, AP, PC, CP), and research areas of authors as labels.
Aminer is academic heterogeneous graph dataset. It contains three types of nodes (paper (P), author (A), reference (R)), four types of edges (PA, AP, PR, RP), and categories of papers labels.
Freebase is huge knowledge heterogeneous graph dataset. It contains eight types of nodes (book (B), film (F), location (L), music (M), person (P), sport (S), organization (O), business (U)), 62 types of edges, and categories of books labels.
Amazon-Phote and Amazon- Contutersss are two co-purchase homogeneous graph datasets. They are two networks extracted from Amazon's 's co-purchase data. Nodes are products, and edges denote that these products were often bought together. Products are categorized into several classes by the product category.
Coauthor-CC and Coauthor-Physie are co-author homogeneous graph datasets. They are two networks extracted from the Microsoft Academic Graph (22). Nodes are authors, and edges denote a collaboration of authors. Authors are categorized into several classes by research fields.
D.2 COMPARISON METHODS
The comparison methods include eleven heterogeneous graph methods and twelve homogeneous
graph methods. Heterogeneous graph methods include Mp2vec (Dong et al. (017). HAN (Wang
al. 2019), HGT (Hu al. 2020), DMGI (Park al. 2020), DMGlattn (Park et al. 2020 HDMI
21

---

Published as a conference paper at ICLR 2024
Table 4: The characteristics of all comparison methods.
Methods
Heterogeneous
Homogeneous
Semi-sup
Self-sup/unsu
Meta-path
DeepWalk (2014)
GCN (2017)
GAT (2018)
DGI (2019)
GMI (2020)
MVGRL (2020)
GRACE (2020)
GCA (2021)
GIC (2021)
G-BT (2022)
COSTA (2022)
DSSL (2022)
Mp2vec (2017)
HAN (2019)
HGT (2020)
DMGI (2020)
DMGlatt (2020)
v
HDMI (2021)
HeCo (2021)
HGCML (2023)
V
CPIM (2023)
HGMAE (2023)
DMG (2023)
HERO (ours)
(Jing et al.) 2021), HeCo (Wang et al. 202T), HGCML (Wang et al. 2023 CPIM (Mo et al.
2023b), HGMAE (Tian al. (003), and DMG (Mo et al. 2023a). Homogeneous graph methods
include GCN (Kipf & Welling 2017). GAT (Velickovic et al. 2018)), DeepWalk (Perozzi et al.
2014), DGI (Velickovic al. 2019), GMI (Peng et al.. 2020), MVGRL (Hassani & Khasahmadi
2020), GRACE (Zhu et al. 20206), GCA (Zhu et al. 2021). GIC (Mavromatis & Karypis 2021).
G-BT Bielak et al. 2022), COSTA (Zhang et 2022b). and DSSL (Xiao al. 2022). ). The
characteristics of all methods are listed in Table 4 where "Heterogencous" and "Homogeneous"
indicate the methods designed for the heterogeneous graph and homogeneous graph, respectively. "Semi-sup", and
indicate that the method conducts semi- semi-supervised learning, and hutu-suuprrviseduuuuerrviii learning, respectively. "Meta-path" indicates that the method requires pre-defined meta-paths during the training process.
Table 5: Similarity search performance (i.e., Sim@5 and Sim @ 10) on heterogeneous graph datasets.
ACM
Yelp
DBLP
Aminer
Method
Sim@
Sim@ 10
Sim@5
Sim@ 10
Sim@5
Sim@ 10
Sim@5
Sim@ 10
Deep Walk
78.7+0.2
76.7 76.730..3
73.6:10.4
72.1
85.710.2
84.610.3
67.1
65.910.6
GCN
86.8+10.3
84.910.3
85.1.0.5
83.910.4
88.2+0.11
87.410.2
77.410.4
75.2:0.55
GAT
86.5+0022
85.4+0..4
85.3+0..3
84.2+t0.5
90.6+t0.3
90.2++0.3
06.
74.6+t0.6
Mp2vec
82.3:10.4
81.2.1.55
79.111.6
77.910.7
87.1
85.810.6
71.810.3
68.6.10.4
HAN
87.2+00.2
85.6+0.3
87.5+0.77
87.4+:..4
89.3+0.3
89.3+0.5
78.240.4
78.0+
HGT
90.1+0.5
90.0+0.6
88.6+0.7
88.0+0.5
90.3+0.4
89,9+0.7
79.2+0.7
78.3+0.6
DMGI
89.8+0..3
89.1=00.1
83.2+0.55
82.5+0.7
91.3++0.4
91.0=t0.3
76.3+0.8
73.3+00.7
DMGlattn
90.1-10.2
88.910.4
80.740.4
79.610.2
89.0.1.22
87.910.4
72.8:10.6
71.5.0..5
HDMI
89.8-10.4
89.0+0.2
83.2-1..4
79.810.7
91.244..
90.94.022
78.5.10.5
77.140.7
HeCo
89.6-10.2
88.740.2
86.410.3
85.140.4
90.94002
90.610.1
82.940.7
81.740.55
HGCML
89.5+0.5
89,010.6
87.110.6
85.810.7
91.31004
90.810.6
81.710.6
80.5+0.4
CPIM
89.6:10.6
88.810.4
87.910.77
86,810.5
91.1
91.011.3
82.110.6
80,910.8
HGMAE
89.5+0.33
88.910.5
88.110.8
86,910.7
90.910.4
90.710.5
80.510.4
79,510.6
DMG
89.2+0.66
88.440.5
88.340.6
87.5+0.3
91.4+0.5
91.0++0.7
83.140.4
81.8+0.5
HERO
89.7+0.66
89,2+0.4
89.7+0.3
88.9+0.5
92.1+0.7
91.9+0.5
86.8+0.3
85.1+0.6
22

---

Published as a conference paper at ICLR 2024
Table 6: Classification performance (i.e., Macro-Fl and Micro-FI) of different components of the objective function on heterogeneous graph datasets.
ACM
Yelp
DBLP
Aminer
Method
Macro-F1
Micro-F1
Macro-F1
Micro-FI
Macro-F1
Micro-FI
Macro-F1
Micro-Fl
w/o Leon
89.910.5
89.8.0.77
91.2:0.5
91.0.10.3
88.0.10.6
89.110.5
74.610.7
84.0.10.9
w/o Lape
90.4.10.3
90.311..5
91.2.1044
88.510.7
92.110.6
93.110.5
74.1100.7
83.110..5
HERO
92.2.10.5
92.1110.7
0.7
92.3.10.6
9..8.1066
94.4.10.4
75.1
84.5.10.9
D.3 MODEL ARCHITECTURES AND SETTINGS
As described in Section the proposed method employs the MLP (i.e., 96) and the closed-form solution of the self-expressive matrix S" to obtain the homophilous representations Z. Moreover, the proposed method employs the heterogeneous encoder (i.e., fo) to obtain heterogeneous representations Z. After that, the proposed method employs projection head Pp and to map the homophilous representations and heterogeneous representations into latent spaces. In the proposed method, projection head P3 and are simply implemented by the one linear layer, followed by a ReLU activation. In addition, the proposed method employs transformation us avoid directly aligning the homophilous and heterogeneous representations. The transformation ug is also implemented by the one linear layer, followed by ReLU activation. Finally, In the proposed method, all parameters were optimized by the Adam optimizer (Kingma & Ba 2015) with an initial learning rate. Moreover, We use early stopping with a patience of 30 to train the proposed SHGL model. In all experiments, we repeat the experiments five times for all methods and report the average results.
D.4 EVALUATION PROTOCOL
We follow the evaluation in previous works (Jing et al., 2021; Pan & Kang 2021 Zhou et al. 2022)
conduct node classification and similarity search as remi-supervisedd and unsupervised downstream tasks, respectively. Specifically, we first pre-train models with unlabeled data in a self-supervised manner and output learned node representations. After that, the resulting representations can be used for different downstream tasks. For the node classification task, we train a simple logistic regression classifier with fixed iteration number, and evaluate the effectiveness of all methods with Micro-F1 and Macro-FI scores, For the similarity search task, we first compute the cosine similarity between the representations of all node pairs, then calculate the ratio possessing the same label within the top-5 and top-10 10 most similar node pairs (Sim@) @5 5 and Sim @ 10).
E ADDITIONAL EXPERIMENTS
This section provides some additional experimental results to support the proposed method
(i.e., HERO), including experiments on the large-scale dataset in Section E.I ablation studies
in Section E.8 visualization of self-expressive matrix in Section E.9 parameter analysis in
Section E. 10
comparison experiments on the similarity search task in Table
Table 7: Classification performance (i.e., Macro-F1 and Micro-F1) of different representations on heterogeneous graph datasets.
ACM
Yelp
DBLP
Aminer
Method
M1
Micro-F1
Macro-F1
Micro-F1
Macro-F1
Micro-F1
Macro-F1
Micro-F1
w/o Z
91.2.10.7
91.110.5
91.1100.5
91.3110.6
93.1100.7
93.7110.6
72.4.0.55
81,9
w/o
87.810066
87.8.10.4
89.310.4
89.410.55
74.6.10.5
75.5.10.7
58.5100.
63.7.10.6
HERO
92.2.10.5
92.110.7
92.4.0.77
92.310.6
93.8.10.6
94.410.4
75.110,77
84.5-00.9
23

---

Published as a conference paper at ICLR 2024
Table 8: Classification performance (i.e, Macro-Fl and Micro-F1) of the subspace and nearby
neighbors homophily objective function on heterogeneous graph datasets.
ACM
Yelp
DBLP
Aminer
Method
Macro-F1
Micro-F1
Macro-FI
Micro-Fl
Macro-F1
Micro-F1
Macro-F1
Micro-F1
wlo Laub 86.
86.6110.6
91.310.77
90.810.5
92.5.10.6
93.4.008
69.8.10.5
76.8
w/o Lnei
9..4=005
91.3.10.3
91.8.1077
91.3.10.6
93.0.10.4
94.0.10.5
71.410.6
79.810.3
HERO
9.5
92.1110.7
92.410.7
92.3.10.6
93.8.10.6
94.410.4
75.1100.7
84.5.009
Table 9:
Classification performance (i.e., Macro-F1 and Micro-FI) of self-expressive and self- attention mechanisms on heterogeneous graph datasets.
ACM
Yelp
DBLP
Aminer
Method
Macro-F1
Micro-F1
Macro-F1 Micro-F1
Macro-FI
Micro-F1
Macro-F1
Micro-F1
self-attention
88.7=0.8
88,4+00.7
92.0+-05
91.7++0.6
91.2++0.4
73.2+0.7 92.1+0.6
82.1+00.6
self-expressive
9252-0.5
92.1+-0.7
92.4+0.77
9.6
0.6
94.4+0.4 75.1+007
84.5++0.9
E.1 EPFECTIVENESS AND EFFICIENCY ON THE LARGE-SCALE DATASET
The time complexity of the proposed method is linearly related to the sample size, which shows its potential be applied to large-scale datasets. To further verify the effectiveness and efficiency the proposed method on the large-scale dataset, we evaluate the node classification performance, training time, and memory cost the proposed method and all SHGL comparison methods on the huge knowledge graph dataset (i.e., Freebase) and report the results in Figure 5 From Figure 5 we have the observations as follows. First, the proposed method achieves the best classification performance and the minimal training time cost, compared other SHGL comparison methods. This can be attributed to the fact that the time complexity of the proposed method is linearly with the sample size while other comparison methods generally require quadratic time complexity. Second, the proposed method also achieves the minimal memory cost on the huge knowledge graph dataset. This verifies the scalability of the proposed method on the large-scale dataset.
E.2 EPFECTIVENESS OF DIFFERENT COMPONENTS
The proposed method investigates the consistency loss (i.e., Loon), and the specificity loss (i.e., Eepe) extract the consistent information between homophilous and heterogeneous representations and preserve their specific information, respectively. To verify the effectiveness of each component of the objective function in the proposed method, we investigate the performance of all variants of the objective function on the node classification task and report the results in Table 6 Moreover, we also investigate the performance of using only homophilous representations or heterogeneous representations on the node classification task and report the results in Table 7
From Tables 6 and we have the observations as follows. First, the proposed method with the complete objective function achieves the best performance. For example, the proposed method on average improves by 2.7% and 2.0%, compared to the variant method without Ccon and the variant method without Spe respectively, indicating that all components of the objective function are both significant for the proposed method. This suggests that both the consistent information and specific information are important for downstream tasks. Second, the variant method that only applies homophilous representations or heterogeneous representations to downstream tasks obtains inferior performance, compared the proposed method that both uses homophilous representations and heterogeneous representations. This makes sense as the homophilous representations and heterogeneous representations can complement each other, thus benefiting downstream tasks.
E.3 EPFECTIVENESS OF THE
MECHANISM
The proposed method employs the self self-expressive mechanism to capture the homophily from the subspace and nearby neighbors. Actually, the self-expressive mechanism shares similar idea with
24

---

Published as a conference paper at ICLR 2024
Table 10: Classification performance (i.e., Macro-Fl and Micro-FI) of the same and different projection heads on heterogeneous graph datasets.
ACM
Yelp
DBLP
Aminer
Method
Macro-F1
Micro-F1
Macro-FI
Micro-FI
Macro-F1
Micro-F1
Macro-F1
Micro-F1
different pro
89.610.3
89.4-0.7
92.2.1044
92.0.10.5
92.910.77
93.910..5
71.010.4
79.410.5 75.1
same pro
9..
92.1-0.7
92.4
92.3.10.6
93.8.10.6
94.410.4
84.5.1099
Table 11:
Classification performance (i.e, Macro-FI and Micro-F1) of proposed method with the GCN encoder and GAT encoder on homogeneous graph datasets.
Amazon-Photo
Amazon-Computer
Coauthor-CS
Coauthor-Physies
Method
Macro-FI Fl
Micro-FI
Macro-F1
Micro-Fl F1
Macro-F1
Micro-F1
Macro-F1
Micro-F1
GAT
91.2++0.5
92.6-0.7
88.2+-0.4 0..1+-0.3
90.8+-0.6
92.9++044
94.4++0.5
95.8-
GCN
91.8:10.4
93.
85.710.6
88.410.5
90.611..5
93.3:10.6
94.6.10.7
96.0=0.5
the self-attention mechanism, i.e., linearly describing each sample with all samples. We analyzed the major difference between them in Section A.3] To further verify the effectiveness of the self-expressive mechanism, we investigate the performance of the variants methods with the self-expressive and self-attention mechanisms and report the results in Table Obviously, the self-expressive mechanism obtains better performance than the self-attention mechanism on all datasets. The reason can be summarized as follows. First, the self-attention mechanism is restricted to non-negative, while the self-expressive mechanism is not restricted. Therefore, the self seff-expressive matrix is able to capture the positive relationships among nodes within the same class and the negative relationships among nodes from different classes, respectively. Second, the self-expressive matrix in the proposed method encouraged focus on the most relevant neighbors of each sample instead of arbitrary nodes in the self- self-attention mechanism. Therefore, the self-expressive matrix focuses more the neighbors of each node that may come from the same class and less on its faraway nodes that may come from different classes. As result, the effectiveness of the self-expressive mechanism is verified.
E.4 EFFECTIVENESS OF DIFFERENT DIMENSIONS
According to the complexity analysis in Section B.2 the time complexity of the proposed method is O(Nd d + kd). which is highly related to the representation dimension d. Therefore, we conduct an ablation study on the proposed method with different dimensions on four heterogeneous graph datasets and report the results in Figure 6 From Figure 6 we have the observations as follows. First, the proposed method generally achieves the best performance with the dimension set to 128 or 256. Second, the performance of the proposed method is stable the dimension increases. This indicates that the proposed method does not require large dimensions to achieve significant performance. Generally, the representation dimensions satisfy that d² < N, where N is the number of nodes. Therefore, this further verifies the efficiency of the proposed method.
E.5 EFFECTIVENESS OF THE SUBSPACE AND NEARBY NEIGHBOR HOMOPHILY
The proposed method investigates the first term in Eq. to optimize the self-expressive matrix capture the homophily in the same subspace while investigates the second term to enable the self-expressive matrix S to capture the homophily among the node and its nearby neighbors. To verify the effectiveness of two terms in Fq. (3). we denote the first term and the second term as Loub and Crei, respectively. Then we investigate the performance of variants methods with Caub or Lnei only and report the classification results in Table
First, the proposed method considering both the homophily from the subspace and nearby neighbor obtains the best performance. For example, the proposed method on average improves by 2.2% and 4.6%, compared to the variant method that only considers the homophily in the subspace or nearby neighbors. This indicates that both the homophily in the subspace and nearby neighbors are essential for the proposed method and can complement each other, enforcing that the nodes within
25

---

Published as a conference paper at ICLR 2024
Table 12: Classification performance (i.e., Macro-FI and Micro-FI) different fusion mechanisms on heterogeneous graph datasets.
ACM
Yelp
DBLP
Aminer
Method
Macro-Fl
Miero-FI
Macro-F1
Micro-F1
Macro-FI
Micro-FI
Macro-F1
Micro-F1
Average Pooling
92.3.1.22
92.1.0.4
92.2.0.6
92.0.0.5
93.310.3
94.1100.5
74.910.7
83.2.10.6
Max Pooling
91.2.10.55
91.0.0.7
90.9-0.3
90.610.4
91.910.7
92.2.0.8
71.110.5
80.5.10.4
Mix Pooling
92.011..6
91.91.55
91.8-0.5
91.910.7
92.810.6
93.910.4
74.510.7
83.7.10.8
Concatenation
92.2=0.5
92.1=0.7
92.4=0.7
92.3-0.6
93.810.6
94,41044
75.1
84.5+0.9
309
668
SE
SE
2111
DI
260
200
201
ACM
Yelp
Figure and (b) indicate the self-expressive matrix heatmaps of ACM and Yelp datasets reordered by node labels.
the same class have similar representations and benefit downstream tasks. Second, the variant method
with Lad only obtains better performance compared to the variant method with Lnes only. This demonstrates that Lutb in Eq. (3) captures main homophily in the subspace, and Lnes in Eq. (3)
provides some complementary homophily from the nearby neighbors.
E.6 EFFECTIVENESS OF THE PROJECTION HEAD
The proposed method uses the same projection head to map homophilous and heterogeneous repre- sentations into the same latent space. To verify the effectiveness of the projection head, investigate the performance variant methods with the same projection head and different projection head and report the results in Table T0 Obviously, the variant method with the same projection head obtains the best performance. The reason can be attributed to the fact that the same projection head can map homophilous and heterogeneous representations into the same space, so that they are comparable the same latent space.
E.7 EFFECTIVENESS OF THE ENCODER ON HOMOGENEOUS GRAPH
The proposed method replaces the heterogeneous encoder with GCN on homogeneous graph datasets, because the heterogeneous encoder is used to deal with multiple node types in the heterogeneous graph, while there is one node type in the homogeneous graph. To study the impact of the encoders in the proposed method, we conduct experiments for variant methods using different encoders (i.e., GCN and GAT) and report the results in Table II The results indicate that the variant methods with GCN encoder and GAT encoder show similar performance on all datasets, demonstrating that the proposed method is robust to different encoders.
26

---

Published as a conference paper at ICLR 2024
DAGGE
DMGlana
ROMAE
DMG
CPIM
ReCoo (OOM)
BOCML (OOM)
20002
Micro-FI (56)
TIRO
Menory (MD)
50
1900
45
40
HerCo
CPM
Delc
HERO
10
20
30
Training time (minures)
(a) Performance and training time cost
(h) Memory cost
Figure The classification performance, training time cost, and memory cost of the proposed HERO and all SHGL comparison methods on the huge knowledge graph dataset Freebase, where OOM indicates Out-Or-Memory.
Micro-FI (%)
80
ACM
60-
Yelp
DBLP
Aminet
16
32
128
256
512
1024
2048
Dimensions
Figure 6: The classification performance of the proposed method with different representation dimensions on four heterogeneous graph datasets.
E.8 EPFECTIVENESS OF THE FUSION MECHANISM
The proposed method employs simple concatenation mechanism to fuse homophilous represen- tations and heterogeneous representations for downstream tasks. To study the impact of the fusion mechanism, we investigate the performance of the proposed method with different fusion mechanisms (i.e., average pooling (LeCun et al.) 1989), max pooling (Murray & Perronnin, 2014), mix pooling et al. 2014), and concatenation) and report the results Table 12 Obviously, the concatenation mechanism obtains the superior results. This indicates that the proposed method requires only very simple fusion mechanism (e.g., concatenation) achieve good performance, without the need for complex fusion operations.
E.9 VISUALIZATION OF THE SELF-EXPRESSIVE MATRIX
The proposed method employs the self-expressive matrix to capture the homophily from the subspace and nearby neighbors. To verify that the self-expressive matrix indeed captures the homophily the heterogeneous graph, visualize the self-expressive matrix heatmaps of ACM and Yelp datasets in Figure where rows and columms are reordered by node labels. In the heatmaps, the darker a pixel, the larger the value of self-expressive matrix weight. From Figure 4 we observe that the heatmaps exhibit a similar block diagonal structure with the correlation map homophilous representations. This indicates that the self-expressive matrix assigns large weights for nodes from the same class and
27

---

Published as a conference paper at ICLR 2024
small weights for nodes from different classes to describe each node. Therefore, the self-expressive indeed captures the homophily in the heterogeneous graph.
Miere-FI (%)
90
Mani-Fi150
167
70
10.3
10:2
10
Figure 7: The classification performance of the proposed method at different parameter settings (i.e., 7, and A) on the ACM dataset.
E.10 PARAMETER ANALYSIS
In the proposed method, we employ the non-negative parameters (i.e. v), and X) to achieve trade-off between each term of the consistency loss, specificity loss, and the final objective function. To investigate the impact of y, 1). and À with different settings, we conduct the node classification on the ACM datasets by varying the value of parameters the range of [10- ,110 and reporting the results in Figure 7
From Figure 7 the observations can be summarized as follows. First, for the parameters 7 and n. the proposed method consistently achieves significant performance within the range of [10 [10-2 2.00.. Moreover, if the values parameters are too large (e.g., 102) or too small (e.g, < 10 10-3), the proposed method obtains inferior performance. This indicates that each term in the consistency loss and specificity loss is essential for the proposed method. Second, for the parameter A, the proposed method achieves the best results while the value of the parameter is in the range of 110*). This further confirms the importance of both consistency and specificity loss for the proposed method.
F POTENTIAL LIMITATIONS AND FUTURE DIRECTIONS
In this paper, we employ the closed-form solution of the self-expressive matrix to describe and capture the homophily in the heterogeneous graph from the subspace and nearby neighbors, thus benefiting downstream tasks. Actually, it is worth noting that various methods can utilized to capture the interconnections between nodes within the the homophily within the heterogeneous graph. Furthermore, the proposed method relies on node features calculate the feature distance between all pairs of nodes. However, there are cases in the heterogeneous graph where nodes lack features. Although one-hot vectors or structural embedding can be assigned as features to address this issue, we acknowledge the need to develop specific methods tailored for the heterogeneous graph without node features to further enhance effectiveness. We consider these aspects as potential directions for future research.
class, thereby effectively describing
28


Reviews:
[{"review_id": "orvdVVIo0X", "reviewer": "Reviewer_Zp3d", "summary": "This paper addresses challenges in self-supervised heterogeneous graph learning. It points out two main issues: reliance on human-defined meta-paths, and underutilization of graph heterogeneity. To tackle these, the authors propose a method that captures homogeneity without predefined meta-paths using a self-expressive matrix. Additionally, they capture heterogeneity by aggregating information from different node types. Two losses are introduced to ensure consistency between homogeneity and heterogeneity and to preserve task-related information. Theoretical analysis suggests the learned representations effectively capture homogeneity and introduce more task-related information. Experimental results demonstrate the method's superiority across various downstream tasks.", "questions": "please see in the above section", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. I really like the idea of learning homogeneity and heterogeneity without pre-defined meta-paths. It is fresh and interesting.\n\nS2. They theoretically analyze that the learned homogeneous representations exhibit the grouping effect to capture the homogeneity, and considering both homogeneity and heterogeneity introduces more task-related information\n\nS3. The use of a self-expressive matrix to capture homogeneity without predefined meta-paths is a creative solution, potentially reducing the need for expert knowledge. The introduction of consistency and specificity loss functions enhances the model's ability to extract and preserve task-related information.\n\nS4.  Extensive experimental results demonstrate the effectiveness and superiority of the proposed method across various downstream tasks, validating its practical utility. The paper evaluates the proposed method thoroughly on different downstream tasks, showcasing its versatility and robust performance.", "weaknesses": "W1. Complexity and Computational Cost: The paper doesn't extensively discuss the computational complexity and resource requirements of the proposed method, leaving potential concerns about its scalability in large-scale applications, especially the comparison with traditional meta-path-based approaches.\n\nW2. The whole workflow is a little complicated, which may raise concerns about reproducibility. \n\nW3. Some important work missing, the author should discuss them in their paper, including but not limited to Self-supervised Hypergraph Representation Learning for Sociological Analysis. TKDE; Heterogeneous Hypergraph Embedding for Graph Classification. WSDM\n\n\n\nQ1: For section 2.1 (homogeneity), I wonder what the compared results between a message-passing GNN via such a complex self-expressive matrix and the one via graph transformer (learnable self-attention)\n\n\nQ2: I wonder about the scalability and efficiency of real-world datasets.\n\n\nQ3: It seems the whole workflow for node classification is not an end-to-end pipeline, the author first uses their framework to pre-train the model and then they use the learned model for specific node classification. I wonder what's the detailed settings for their task head for the node classification task.\n\n\nQ4. Interpretability: Although the traditional meta-path approach may have some shortcomings as mentioned by the authors,  there is no doubt that these methods are very intuitive for human understanding and explanation. I wonder how the interpretability of the learned representations is. Can the authors provide more insights into how the model's representations can be interpreted, aiding researchers and practitioners in understanding its decision-making process?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6AZFTIkt5y", "reviewer": "Reviewer_6ShR", "summary": "This paper proposes HERO, a self-supervised heterogeneous graph learning method that captures both homogeneity and heterogeneity in the input heterogeneous graph. The authors theoretically analyze the grouping effect provided by capturing homogeneity and the task-related information provided by capturing both homogeneity and heterogeneity. In experiments, HERO outperforms state-of-the-art baselines on both heterogeneous graph datasets and homogeneous graph datasets.", "questions": "Please check the weaknesses above for the main issues. Here are some additional questions/comments.\n1. What is the evaluation protocol for HERO and other baselines? Is it to train a linear classifier (e.g., SVM) on top of (frozen) representations learnt by self-supervised methods?\n2. Metapaths can also associate two nodes with different node types. The definition of metapaths given on page 3 is therefore misleading. It's just many GNN-based methods tend to use the same-type version to construct homogeneous graphs.\n3. The idea of this paper is kind of similar to DHGCN [1], which also explicitly and separately aggregates heterogeneous neighbors (heterogeneity) and metapath-guided same-type neighbors (homogeneity).\n4. Definition 2.1 is currently hard to understand. The sentence \"if the conditions ... hold\" may need to be changed to \"if $|c_{ik}-c_{jk}|\\rightarrow 0$ ($\\forall 1 \\leq k \\leq F^\\prime$) hold for every $v_i, v_j$ satisfying $v_i \\rightarrow v_j$ (i.e., $||x_i-x_j||_2 \\rightarrow 0$)\"\n5. It would be better to also visualize the self-expressive matrix, to show that it indeed captures homogeneity/homophily.\n\n[1] Saurav Manchanda, Da Zheng, George Karypis: Schema-Aware Deep Graph Convolutional Networks for Heterogeneous Graphs. IEEE BigData 2021: 480-489", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea of collectively and explicitly capturing homogeneity and heterogeneity for better SHGL is novel and interesting. The paper is generally well-written without obvious typos or grammatical errors.", "weaknesses": "1. The authors confuse homogeneity and homophily, which impairs the technical soundness of this paper. The definition of homogeneity in the paper is more like homophily. Homogeneity (single node/edge type) and homophily (connected nodes tend to have the same class label) are two different concepts.\n2. Overall, many of the arguments proposed by the authors sound far-fetched. It also seems that the authors take many things for granted.\n    1. Some claims of the authors are not true. For example:\n        - \"previous SHGL methods generally overlook or cannot effectively utilize the heterogeneity.\" Many existing SHGL methods, such as metapath2vec, can capture the heterogeneity. All those GNN-based SHGL methods can also capture both aspects as long as they employ heterogeneous GNN models.\n        - \"metapaths are employed to capture the homogeneity in the heterogeneous graph.\" Actually, many SHGL methods also leverage metapaths to capture the heterogeneity, such as metapath2vec, which applies DeepWalk to the node sequences (containing multiple types of nodes) generated by the pre-defined metapath.\n    2. Unclear benefit of mining heterogeneity. The example given in the introduction section is hard to understand.\n    4. Missing formal definitions for some important terminologies, including homogeneity, heterogeneity, and homogeneity rate.\n    5. Missing derivations or theorem proof for some HERO components, including Equation (10) and Equation (11).\n    6. The observations stated in Section 2.3 come with no supporting evidence.\n3. Missing important baseline: metapath2vec.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xUYO4NK6vy", "reviewer": "Reviewer_h7xr", "summary": "A different insight is given in this paper, compared to previous self-supervised heterogeneous graph representation learning, i.e. capture the homogeneity and heterogeneity without meta-paths. The authors have performed theoretical analysis of the proposed framework from the perspective of grouping effect and downstream tasks.  In addition, sufficient experimental results show that the proposed method achieves best results in several datasets.", "questions": "The proposed method is also evaluated on the homogeneous graph datasets, e.g. Amazon-photo. On homogeneous graph datasets, does the method proposed in this paper not require data augmentation? What are the advantages of the method proposed in this paper compared to popular self-supervised methods on homogeneous graphs that require data augmentation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(1)\tThe paper is clearly written and describes a novel insight and framework for the SHGL, which is rare in similar works.\n\n(2)\tSome theoretical guarantees are given to illustrate the proposed method.\n\n(3)\tSome ablation studies are conducted to verify the effectiveness of each component.", "weaknesses": "(1)\tThe proposed method uses the same projection head to map homogeneous and heterogeneous representations into the same potential space. Do the projection head have to be the same? What are the implications of using two projection heads to map homogeneous and heterogeneous representations separately?\n\n(2)\tThe proposed method employs the closed-form solution of the self-expressive matrix to capture the homogeneity in the heterogeneous graph. It is desirable to clarify the process of deriving the closed-form solution so that the reader can understand the method Section more easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RJ5755IYam", "reviewer": "Reviewer_EpeM", "summary": "This paper proposes to understand the self-supervised heterogeneous graph learning without pre-defined meta-paths from the perspective of homogeneity and heterogeneity. To do this, this paper captures the homogeneity from both the subspace and nearby neighbors as well as to discard pre-defined meta-paths. Moreover, the proposed method further extracts the consistent and specific contents between homogeneous and heterogeneous representations to introduce more task-related information. Experimental results and theoretical analysis demonstrate the superiority of the proposed method.", "questions": "see above", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors provide a deeper insight (i.e., heterogeneity mining and homogeneity mining) into existing self-supervised heterogeneous graph representation learning, which is interesting and reasonable.\n\n2.\tThis paper proposes to use the self-expression matrix rather than traditional meta-paths to capture homogeneity in the heterogeneous graph, which opens up a new possibility for the self-supervised heterogeneous graph learning community to enhance its scalability and effectiveness.\n\n3.\tComprehensive comparison experiments, visualization, and ablation studies are greatly valued. Sufficient datasets from different domains have been used to evaluate the proposed method on different downstream tasks, and the proposed method has been thoroughly discussed through experiments.\n\n4.\tI overviewed the code provided by the authors. It seems well-organized and detailed. Moreover, I tried to run the provided code with the saved checkpoints, and it can easily reproduce the results reported in the paper.", "weaknesses": "1. This paper aims to make the first attempt to simultaneously extract the homogeneity and the heterogeneity without meta-path in the heterogeneous graph. So, what’s the difference and relationship between the homogeneity extraction and the heterogeneity extraction in the heterogeneous graph?\n\n2. As this paper proposes a self-supervised framework for SHGL methods, some recent related works in self-supervised learning can be added in the related work section.\n\n3. The paper needs more proofreading and some typos should be fixed. For example, \nIn the second paragraph of page 4: neighbor set->neighbors set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZsATtJK3m1", "reviewer": "Reviewer_rAzi", "summary": "* This work captures both homogeneity and heterogeneity in the heterogeneous graph without pre-defined meta-paths. \n\n* A self-expressive matrix is used to capture the homogeneity from the subspace and nearby neighbors. Meanwhile, the heterogeneity is captured by aggregating the information of nodes from different types. \n\n* The proposed method extracts the consistent information between homogeneity and heterogeneity and preserves their specific task-related information, leading to the effectiveness. \n\n* The experimental results show that the proposed method achieves SOTA in various datasets, compared to numerous comparison methods.", "questions": "* The idea of capturing the homogeneity from the subspace and nearby neighbors is interesting. Can it be used in other related domains, such as computer vision?\n\n* Why replace the heterogeneous encoder with GCN to implement the proposed method on homogeneous graph datasets? What happens if the heterogeneous encoder is replaced with other similar encoders, e.g., GAT?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* The motivation of the proposed method is clearly stated with the illustrative examples and empirical studies. I really appreciate the examples used in the Introduction. Moreover, the design of the framework is easy to understand.\n\n* The method part is solid, and exploring homogeneity and heterogeneity in the heterogeneous graph without predefined meta-paths makes sense.\n\n* Theoretical analysis verifies that the proposed method captures the homogeneity in the heterogeneous graph and is expected to obtain better performance than previous methods.", "weaknesses": "* A corollary is shown in the Appendix to verify that the representations with both homogeneity and heterogeneity indeed obtain a better downstream task performance than the representations with homogeneity or heterogeneity only. I think this is a strong addition to Theorem 2.3 and should be mentioned in the main text.\n\n* The proposed method seems to yield overall smaller improvements on homogeneous graph datasets than on heterogeneous graph datasets when compared to comparative methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jb0dwdMO9c", "reviewer": "Reviewer_BScY", "summary": "To capture homogeneity and heterogeneity in the heterogeneous graph without pre-defined meta-paths, this work proposes to adaptively learn a self-expressive matrix and employ the heterogeneous encoder to obtain homogeneous and heterogeneous representations. In addition, the proposed method designs the objective function to extract the consistent information between homogeneous representations and heterogeneous representations and to maintain their specific information in different latent spaces. Moreover, the authors theoretically demonstrate the effectiveness of the proposed method and support the claims made in this paper.", "questions": "a)The effectiveness of the proposed HERO and other comparison methods are evaluated through two downstream tasks, namely, node classification and similarity search. What are the significant distinctions between these tasks?\n\nb)Theorem 2.3 does not constrain the type of downstream tasks. Does it mean that the proposed method is expected to achieve better performance on different downstream tasks, compared to the methods considering the homogeneity only?\n\nOverall, this paper holds considerable value and potential, and I will increase my rating if the weaknesses and questions can be addressed and discussed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a)Idea in this work is overall novel and attractive, and it will inspire researchers in related fields to explore new methods to capture the homogeneity in the heterogeneous graph without meta-paths, which induces inconveniences and large computation costs.\n\nb)Extensive experimental results demonstrate the effectiveness of the proposed method on both homogeneous and heterogeneous graph datasets in terms of different downstream tasks.\n\nc)Claims in this paper are supported well. That is, the learned homogeneous representations are demonstrated to exhibit the grouping effect to capture the homogeneity, and considering both homogeneity and heterogeneity introduces more task-related information to benefit downstream tasks.", "weaknesses": "a)Not clear why the concatenation mechanism is directly employed to fuse homogeneous representation and heterogeneous representations to obtain final representations for downstream tasks. What would be the impact on the performance of the method if other fusion methods were used, such as average pooling? The authors could add such ablation studies.\n\nb)It would be better to add some details of experiments. For example, the detailed information of all datasets used in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)