Title: Revisiting Self-Supervised Heterogeneous Graph Learning from Spectral Clustering Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: Heterogeneous graph

Spectral clustering

Self-supervised learning
Abstract: Self-supervised heterogeneous graph learning (SHGL) has shown promising potential in diverse scenarios. However,  while existing SHGL methods share a similar essential with clustering approaches, they encounter two significant limitations: (i) noise in graph structures is often introduced during the message-passing process to weaken node representations, and (ii) cluster-level information may be inadequately captured and leveraged, diminishing the performance in downstream tasks. In this paper, we address these limitations by theoretically revisiting SHGL from the spectral clustering perspective and introducing a novel framework enhanced by rank and dual consistency constraints. Specifically, our framework incorporates a rank-constrained spectral clustering method that refines the affinity matrix to exclude noise effectively. Additionally, we integrate node-level and cluster-level consistency constraints that concurrently capture invariant and clustering information to facilitate learning in downstream tasks. We theoretically demonstrate that the learned representations are divided into distinct partitions based on the number of classes and exhibit enhanced generalization ability across tasks. Experimental results affirm the superiority of our method, showcasing remarkable improvements in several downstream tasks compared to existing methods.

Revisiting Self-Supervised Heterogeneous Graph Learning from Spectral Clustering Perspective

Abstract

Self-supervised heterogeneous graph learning (SHGL) has shown promising potential in diverse scenarios. However, while existing SHGL methods share a similar essential with clustering approaches, they encounter two significant limitations: (i) noise in graph structures is often introduced during the message-passing process to weaken node representations, and (ii) cluster-level information may be inadequately captured and leveraged, diminishing the performance in downstream tasks. In this paper, we address these limitations by theoretically revisiting SHGL from the spectral clustering perspective and introducing a novel framework enhanced by rank and dual consistency constraints. Specifically, our framework incorporates a rank-constrained spectral clustering method that refines the affinity matrix to exclude noise effectively. Additionally, we integrate node-level and cluster-level consistency constraints that concurrently capture invariant and clustering information to facilitate learning in downstream tasks. We theoretically demonstrate that the learned representations are divided into distinct partitions based on the number of classes and exhibit enhanced generalization ability across tasks. Experimental results affirm the superiority of our method, showcasing remarkable improvements in several downstream tasks compared to existing methods.

1 Introduction

Self-supervised heterogeneous graph learning (SHGL) aims to effectively process diverse types of nodes and edges in the heterogeneous graph, producing low-dimensional representations without the need for human annotations [72, 71, 25]. Thanks to its remarkable capabilities, SHGL has attracted significant interest and has been utilized in a broad array of applications, including recommendation systems [44, 12], social network analysis [45, 9], and molecule design [68, 59].

Existing SHGL methods can be broadly classified into two groups, i.e., meta-path-based methods and adaptive-graph-based methods. Meta-path-based methods typically utilize pre-defined meta-paths to explore relationships among nodes that may share the same label in the heterogeneous graph [18, 74]. However, building meta-paths requires extensive prior knowledge and incurs additional computation costs [69]. To address these drawbacks, adaptive-graph-based methods dynamically assign significant weights to node pairs likely to share the same label, using the adaptive graph structure rather than traditional meta-paths [30]. Both groups of SHGL methods facilitate message-passing among nodes within the same class, either through meta-path-based graphs or adaptive graph structures. As a result, this process minimizes intra-class differences and promotes a clustered pattern in the learned representations, aligning these methods closely with conventional clustering techniques.

…

Z

q



Q

~

~

Z

Argmax

Matmul

Z

.3

.4

.9 -.4

-.4 .8

.4

.9

.3

.3

.4

.9 -.4

-.4 .8

.4

.9

.3

· ·

· ·

· ·

.3

.4

.9 -.4

-.4 .8

.4

.9

.3

· ·

· ·

· ·

· · ·

H

Q

Q

~

Y

Heterogeneous

f



Encoder

Multi-Layer

g



Perceptron

…

p



Projection

head

Orthogonal

Layer

Orthogonal

Layer

Orthogonal

Layer

Orthogonal

Layer

Projection

q



head

Projection

q



head

Share

sp

cc

S

ˆ Y

ˆ Q

nc

Despite the effectiveness of previous SHGL methods, they encounter two limitations. First, previous methods conduct message-passing relying on meta-path-based graphs and adaptive graph structures, which inevitably include noise, i.e., connections among nodes from different classes [18, 58]. Consequently, such noise compromises the identifiability of node representations after the message-passing process. Second, while previous methods exhibit clustering characteristics, they typically emphasize the node-level consistency only, neglecting to capture and leverage the cluster-level information effectively [57, 30]. This may not fully exploit the potential benefits of clustering for representation learning, thereby diminishing the performance of downstream tasks.

Based on the above analysis, it is feasible to analyze previous SHGL methods from a clustering perspective thanks to their close connection to clustering techniques and further optimize the graph structures to mitigate noisy connections as well as harness the cluster-level information to enhance previous SHGL. To achieve this, there are three key challenges, i.e., (i) How to formally understand previous SHGL methods from the clustering perspective? (ii) With insights from the clustering analysis, how to learn an adaptive graph structure that effectively captures intra-class connections while filtering out inter-class noise? (iii) How to enable the effective incorporation of cluster-level information in the heterogeneous graph to boost the performance of downstream tasks?

In this paper, we address the outlined challenges by first theoretically revisiting previous SHGL methods from a clustering perspective and then introducing a novel framework, termed Spectral Clustering-inspired HeterOgeneOus graph Learning (SCHOOL for short), that incorporates rankconstrained spectral clustering and dual consistency constraints, as depicted in Figure 1. Specifically, we start by proving that existing SHGL can be reformulated as spectral clustering with an additional regularization term under the assumption of orthogonality, thus addressing challenge (i) and laying the foundational theory for our approach. Next, to tackle challenge (ii) , we propose an efficient spectral clustering technique that includes a rank constraint on the affinity matrix, aiming to effectively mitigate noisy connections among different classes. Furthermore, to resolve challenge (iii) , we design dual consistency constraints at both node and cluster levels to capture invariant and clustering information, respectively, which reduces the intra-cluster differences and enhances the performance of downstream tasks. Finally, theoretical analysis indicates that the learned representations are divided into distinct partitions corresponding to the number of classes, and are demonstrated superior generalization ability compared to those derived from previous SHGL methods.

Compared to previous SHGL works, our contributions can be summarized as follows:

To the best of our knowledge, we make the first attempt to theoretically revisit previous SHGL methods from the spectral clustering perspective in a unified manner.

We adaptively learn a rank-constrained affinity matrix to mitigate noisy connections inherent in previous SHGL methods. Moreover, we introduce dual consistency constraints to capture both invariant and clustering information to enhance the effectiveness of our method.

We theoretically demonstrate that the proposed method divides the learned representations into distinct partitions based on the number of classes, instead of dimensions in previous SHGL methods. Furthermore, the representations obtained by this method exhibit enhanced generalization ability compared to those derived from previous SHGL methods.

We experimentally manifest the effectiveness of the proposed method across a variety of downstream tasks, using both heterogeneous and homogeneous graph datasets, compared to numerous state-of-the-art methods.

2 Method

Notations. Let G = ( V , E , X , T , R ) represent a heterogeneous graph, where V and E indicate set of nodes and set of edges, respectively. X = { x i } n i =1 denotes the matrix of node features, where n indicates the number of nodes. Moreover, T and R indicate set of node types and set of edge types, respectively. Given the heterogeneous graph G , most existing SHGL methods utilize meta-paths or adaptive graph structures to explore connections among nodes within the same class, thus exhibiting the characteristics of clustering and obtaining discriminative representations. To gain a deeper insight of previous SHGL methods, we first propose to revisit them from a clustering perspective as follows.

2.1 Revisiting Previous SHGL Methods from Spectral Clustering

As mentioned above, previous SHGL methods tend to conduct clustering implicitly, relying on meta-path-based graphs or adaptive graph structures. For example, given an academic heterogeneous graph with several node types ( i.e., paper, author, and subject), for the meta-path-based methods, if two papers belong to the same class, there may exist a meta-path 'paper-subject-paper' to connect them ( i.e., two papers are grouped into the same subject). Similarly, for the adaptive-graph-based methods, when two papers belong to the same class, the adaptive graph structures likely assign large weights to connect them. Therefore, representations of nodes within the same class will be close to each other after the message-passing process, thus implicitly presenting a clustered pattern.

Based on the above observation, actually, we can further theoretically understand previous SHGL methods from the clustering perspective. To do this, we first give the following definition.

Definition 2.1. (Spectral Clustering) Given the Laplacian matrix L , the optimization problem of the spectral clustering can be described as follows:

where L = D -W , W ∈ R n × n is a data similarity matrix, D is a diagonal matrix whose entries are column sums of W , H ∈ R n × d is a representations matrix, Tr( · ) indicates the matrix trace, and I indicates the identity matrix.

According to Definition 2.1, for both meta-path-based methods [31, 57, 58] and adaptive-graph-based SHGL methods [30], we then have Theorem 2.2, whose proof can be found in Appendix C.1.

Theorem 2.2. Assume the learned representations H are orthogonal, optimizing previous meta-pathbased and adaptive-graph-based SHGL methods is equivalent to performing spectral clustering with additional regularization, i.e.,

where R ( · ) indicates the regularization term, ˆ L indicates the Laplacian matrix of the meta-path-based graph or the adaptive graph structure.

Theorem 2.2 reveals the connection between previous SHGL and the spectral clustering as well as indicates that previous SHGL heavily relies on the Laplacian matrix of meta-path-based graph or adaptive graph structure. Moreover, based on Theorem 2.2, we can further bridge previous SHGL and the graph-cut algorithm [54] as follows, whose proof can be found in Appendix C.2.

Theorem 2.3. Under the same assumption in Theorem 2.2, optimizing previous meta-path-based and adaptive-graph-based SHGL methods is approximate to performing the RatioCut( V 1 , . . . , V d ) algorithm that divides the learned representations into d partitions { V 1 , . . . , V d } , i.e.,

where d indicates the dimension of representations H .

Theorem 2.3 further indicates that previous SHGL methods divide the learned representations into d partitions, where d is generally much larger than the number of classes. Therefore, Theorem 2.3 connects the traditional graph-cut algorithm with existing SHGL methods, which requires custom analysis. As a result, we theoretically revisit previous SHGL methods from spectral clustering as well as graph-cut perspectives and build the connections between them, thus solving the challenge (i).

2.2 Rank-Constrained Spectral Clustering

Based on the connections between previous SHGL methods and the spectral clustering as well as the graph-cut algorithm, we have the observations as follows. First, according to Theorem 2.2, previous SHGL methods conduct spectral clustering based on the Laplacian matrix of meta-path-based graph or adaptive graph structure, which may not guarantee optimality and could potentially contain noisy connections, thus affecting the spectral clustering. Second, according to Theorem 2.3, previous SHGL methods conduct the graph-cut to divide the learned representations into d partitions, which are generally not equal to the number of classes c . As a result, optimizing previous SHGL methods becomes a hard or even error problem, and the learned representations can not be clustered well. Therefore, it is intuitive to mitigate noise in the adaptive graph structure as well as divide the learned representations into exactly c partitions to improve existing SHGL methods.

Specifically, in this paper, we propose to learn an adaptive affinity matrix with the rank constraint to mitigate noisy connections as much as possible. To do this, we first employ the Multi-Layer Perceptron (MLP) as the encoder g ϕ ∈ R f × d 1 to obtain the semantic representations H by:

where f and d 1 are the dimensions of node features and representations, respectively, and σ is the activation function. After that, we propose to learn an adaptive affinity matrix S ∈ R n × n based on the semantic representations. Intuitively, in a uncorrelated representations subspace, a smaller distance ∥ h i -h j ∥ 2 2 between semantic representations should be assigned a larger probability s ij . Therefore, it is a natural approach to learn the affinity matrix S by:

where α is a non-negative parameter. In Eq. (5), the first term encourages the affinity matrix to assign large weights to node pairs with small distances. Moreover, the second term avoids the trivial solution that only the nearest node can be the neighbor of v i with probability 1. However, similar to previous SHGL methods, usually the affinity matrix learned by Eq. (5) can not reach the ideal case ( i.e., having no noisy connections among different classes and containing exactly c connected components). As a result, the noisy connections in the affinity matrix may induce a negative interference during the message-passing process. To solve this issue, we first introduce the following lemma in [32].

Lemma 2.4. The multiplicity c of the eigenvalue 0 of the Laplacian matrix L S is equal to the number of connected components in the affinity matrix S .

Lemma 2.4 indicates that if the rank of L S equals to n -c , then the affinity matrix S contains exactly c connected components to achieve the ideal scenario, where L S = D -S + S T 2 , and D is the degree matrix of S + S T 2 . Based on Lemma 2.4, we can solve the above issue by adding the rank constraint on the affinity matrix, i.e., enforcing the smallest c eigenvalues of L S to be 0:

where τ i ( L S ) is the i -th smallest eigenvalue of L S and τ i ( L S ) ≥ 0 because L S is positive semidefinite. Moreover, according to Ky Fan's Theorem [5], the constraint in Eq. (6) can be rewritten in the spectral clustering form as follows (derivation listed in Appendix C.5).

where F ∈ R n × c is formed by part eigenvectors of L S after the eigendecomposition. Therefore, based on Eq. (7), we can add the rank constraint on the affinity matrix and reformulate Eq. (5) as:

where β is a non-negative parameter. Eq. (8) can be solved by applying the alternating optimization approach. Specifically, when S is fixed, Eq. (8) becomes min F T F = I ∑ n i,j =1 ∥ f i -f j ∥ 2 2 s ij = 2 min F T F = I Tr( F T L S F ) , whose optimal solution F is the eigenvectors of L S corresponding to the c smallest eigenvalues. When F is fixed, denote d ij = ∥ h i -h j ∥ 2 2 + β ∥ f i -f j ∥ 2 2 , Eq. (8) can be written in the vector form, i.e.,

According to the Lagrangian function of Eq. (9) and the KKT condition [3], we can obtain the closed-form solution of the element s ij in the affinity matrix, i.e.,

where λ is a non-negative parameter, and ( · ) + indicates max {· , 0 } . In practice, a sparse affinity matrix usually obtains better performance and reduces computation costs. Therefore, we only calculate s ij between node v i and its k nearest neighbors. Then parameters α and λ can be further determined, details listed in Appendix C.6.

However, when fixing S and optimizing F in Eq. (8), computation costs of obtaining eigenvectors F remain prohibitive due to the cubic time complexity of the eigendecomposition. To address this issue, we propose to replace the eigendecomposition with a projection head and orthogonal layer [42]. Specifically, we first employ the projection head p φ ∈ R d 1 × c to map semantic representations to the cluster assignment space P ∈ R n × c , i.e.,

where σ is the activation function. After that, we employ an orthogonal layer to derive the orthogonal cluster assignment matrix Y ∈ R n × c (orthogonal derivation listed in Appendix C.7), i.e.,

where R is an upper triangular matrix obtained from the QR decomposition [6] ( i.e., P = ER and E T E = I ) on the full rank P . Similarly, we can also implement the orthogonal layer to achieve the uncorrelation ( i.e., H T H = I ) on the representations subspace.

As a result, the projection head and the orthogonal layer act as a linear transformation to achieve the orthogonality constraint on Y . To replace the eigendecomposition, the cluster assignment matrix Y should further fit the eigenvectors F in Eq. (8). To do this, we design a spectral loss L sp to optimize the parameters in p φ to simulate the spectral clustering of the third term in Eq. (8), i.e.,

where γ is a non-negative parameter, H ( Y ) = -∑ c i =1 P ( y i ) log P ( y i ) is the entropy of cluster assignment probabilities P ( y i ) = 1 n ∑ n j =1 y i j , and y i j indicates the i -th column and j -th row of y . According to Eq. (7), the first term in Eq. (13) simulates the spectral clustering to enforce the learned cluster assignment matrix Y to approximate eigenvectors F , and the second term is a widely used regularization term [1, 14] to avoid the trivial solution that most nodes are assigned to the same cluster. Therefore, when S is fixed, we optimizing Y to approach the optimal F by achieving orthogonality

with Eq. (12), and fitting eigenvectors F with Eq. (13). As a result, this reduces the cubic time complexity of eigendecomposition to O ( nd 2 + nc 2 + nkd + nkc + c 3 + d 3 ) , where d 2 , c 2 < n , thus is linear to the sample size, details are shown in Appendix B.2. Finally, the proposed method still optimizes the affinity matrix S and eigenvectors F in Eq. (8) in an alternating approach, i.e., fix F and then obtain the closed-form solution of S , and fix S and then optimize parameters ( i.e., g ϕ and p φ ) to update Y to approach the optimal F .

Therefore, the proposed method obtains the affinity matrix with exactly c connected components to mitigate noisy connections in an effective and efficient way. Then we can obtain the node representations Z = SH , which is expected to conduct message-passing among nodes within the same class. Moreover, we can bridge the connections between the proposed method and the spectral clustering as well as the graph-cut algorithm as follows, whose proof can be found in Appendix C.3.

Theorem 2.5. Optimizing the spectral loss L sp leads to performing the spectral clustering based on the affinity matrix S with c connected components and conducting RatioCut ( V 1 , . . . , V c ) algorithm to divide the learned representations into c partitions, i.e.,

Theorem 2.5 indicates that the proposed method conducts the spectral clustering as previous SHGL methods, but is performed on an affinity matrix with exactly c connected components (verified in Section 3.2.3), thus mitigating noisy connections from different classes and solving the challenge (ii). Moreover, the proposed method divides the learned representations into c partitions, which is a better optimization goal than previous SHGL methods to obtain discriminative representations.

2.3 Dual Consistency Constraints

The message-passing among nodes within the same class reduces intra-class differences and enhances node representations Z . Meanwhile, the message-passing among nodes from different types also contributes to obtaining task-related contents and benefits downstream tasks [69]. To do this, we propose to aggregate the information of nodes from different types in the heterogeneous graph with a heterogeneous encoder f θ ∈ R f × d 1 .

Specifically, for the node v i , we concatenate the information of itself and its relevant one-hop neighbors ( i.e., nodes of other types) based on edge types in R , and then derive the heterogeneous representations ˜ Z by:

where σ is the activation function, |R| indicates the number of edge types, N i,r indicates the set of one-hop neighbors of node v i based on the edge type r ∈ R , f θ ( · ) indicates the linear transformation, and ·||· indicates the concatenation operation. Therefore, the heterogeneous representations ˜ Z aggregate the information of nodes from different types to introduce more task-related contents.

Given node representations Z and heterogeneous representations ˜ Z , most previous SHGL methods utilize the node-level consistency constraint ( e.g., Info-NCE loss [36]) to capture the invariant information between them and enhance the effectiveness [57, 30]. In addition, according to Theorem 2.2, previous SHGL methods actually perform spectral clustering to learn node representations. However, previous SHGL methods fail to utilize the cluster-level information outputted by the spectral clustering, thus weakening the downstream task performance. To solve this issue, we design dual consistency constraints to capture the invariant information as well as the clustering information between Z and ˜ Z .

Specifically, we first employ a projection head q γ ∈ R d 1 × d 2 to map both Z and ˜ Z into the same latent space, i.e., Q = q γ ( Z ) and ˜ Q = q γ ( ˜ Z ) , where d 2 is the projected dimension. Then we follow previous works [57, 58] to design a node-level consistency constraint to capture the invariant information between Q and ˜ Q , i.e.,

where C = Q T Q + ˜ Q T ˜ Q , and η is a non-negative parameter. Similar to previous works, the first term in Eq. (16) enforces representations in ˜ Q agree with the corresponding representations in Q , thus capturing the invariant information between them. The second term enables different dimensions of Q and ˜ Q to uniformly distribute over the latent space to avoid collapse.

In addition to the node-level consistency constraint, we further design the cluster-level consistency constraint to capture the clustering information from the cluster assignment matrix Y . To do this, we first obtain the cluster indicator matrix ˆ Y based on the cluster assignment matrix Y , i.e., ˆ Y = argmax( Y ) . After that, we conduct average pooling on node representations that possess the same cluster indicator to obtain the j -th cluster representation ˆ q j , i.e.,

where V j indicates the set of nodes whose cluster indicators equal to j , and | V j | indicates the number of nodes in V j . Then we design a cluster-level consistency constraint on cluster representations ˆ Q and projected representations ˜ Q to capture the clustering information, i.e.,

where ˆ q y i indicates the cluster representation whose label equals to y i . Eq. (18) enables the projected representation ˜ q i and the cluster representation ˆ q y i to align each other. As a result, representations capture the clustering information based on cluster indicators and reduce intra-cluster differences to improve the performance of downstream tasks, thus solving challenge (iii).

We integrate the spectral loss in Eq. (13), the node-level consistency constraint in Eq. (16), with the cluster-level consistency constraint in Eq. (18) to have the objective function as:

where µ and δ are non-negative parameters. Finally, we concatenate node representations Z with heterogeneous representations ˜ Z to obtain representations for downstream tasks. Actually, for the learned representations, we have the following Theorem, whose proof can be found in Appendix C.4.

Theorem 2.6. The proposed method with dual consistency constraints achieves a lower boundary of the model complexity C and a higher generalization ability boundary G than previous SHGL with the node-level consistency constraint only, i.e.,

where inf( · ) and sup( · ) indicates lower bound and upper bound, respectively.

Theorem 2.6 indicates that the representations learned by the dual consistency constraints can be theoretically proved to exhibit superior generalization ability than the representations learned by previous SHGL methods with the node-level consistency constraint only, thus are expected to perform better in different downstream tasks (verified in Section 3.2).

3 Experiments

In this section, we conduct experiments on both heterogeneous and homogeneous graph datasets to evaluate the proposed method in terms of different downstream tasks ( i.e., node classification and node clustering), compared to both heterogeneous and homogeneous graph methods. Detailed settings are shown in Appendix D, and additional results are shown in Appendix E.

3.1 Experimental Setup

3.1.1 Datasets

The used datasets include four heterogeneous graph datasets and two homogeneous graph datasets. Heterogeneous graph datasets include three academic datasets ( i.e., ACM [56], DBLP [56], and Aminer [11]), and one business dataset ( i.e., Yelp [27]). Homogeneous graph datasets include two sale datasets ( i.e., Photo and Computers [43]).

3.1.2 Comparison Methods

The comparison methods include eleven heterogeneous graph methods and twelve homogeneous graph methods. The former includes two semi-supervised methods ( i.e., HAN [56] and HGT [13]), one traditional unsupervised method ( i.e., Mp2vec [4]), and eight self-supervised methods ( i.e., DMGI [38], DMGIattn [38], HDMI [18], HeCo [57], HGCML [58], CPIM [31], HGMAE [48], and HERO [30]). The latter includes two semi-supervised methods ( i.e., GCN[20] and GAT [50]), one traditional unsupervised method ( i.e., DeepWalk [41]), and nine self-supervised methods, ( i.e., DGI [51], GMI [40], MVGRL [8], GRACE [75], GCA [76], G-BT [2], COSTA [70], DSSL [61], and LRD [63]).

For a fair comparison, we follow [4, 56, 27, 28] to select meta-paths for previous meta-path-based SHGL methods. Moreover, we follow [29] to implement homogeneous graph methods on heterogeneous graph datasets by separately learning the representations of each meta-path-based graph and further concatenating them for downstream tasks. In addition, we replace the heterogeneous encoder f θ with GCN to implement the proposed method on homogeneous graph datasets because there is only one node type in the homogeneous graph. Moreover, we follow previous works [76] to generate two different views for the homogeneous graph by removing edges and masking features. The code of the proposed method is released at https://github.com/YujieMo/SCHOOL .

3.2 Results Analysis

3.2.1 Effectiveness on Heterogeneous and Homogeneous Graph

We first evaluate the effectiveness of the proposed method on the heterogeneous graph datasets and report the results of node classification and node clustering in Table 1 and Appendix E, respectively. Obviously, the proposed method obtains better performance on both node classification and node clustering tasks than comparison methods.

Specifically, first, for the node classification task, the proposed method consistently outperforms the comparison methods by large margins. For example, the proposed method on average, improves by 1.1%, compared to the best SHGL method ( i.e., HERO), on four heterogeneous graph datasets. The reason can be attributed to the fact that the proposed method adaptively learns a rank-constrained affinity matrix to mitigate noisy connections among different classes, thus reducing intra-class differences. Second, for the node clustering task, the proposed method also obtains promising improvements. For example, the proposed method on average, improves by 3.1%, compared to the best SHGL method ( i.e., HGMAE), on four heterogeneous graph datasets. This demonstrates the superiority of the proposed method, which simulates the spectral clustering with the spectral loss and conducts the cluster-level consistency constraint to further utilize the clustering information. As a result, the effectiveness of the proposed method is verified on different downstream tasks.

We further evaluate the effectiveness of the proposed method on homogeneous graph datasets and report the results of node classification in Appendix E. We can observe that the proposed method also achieves competitive results on the homogeneous graph datasets compared to other homogeneous graph methods. For example, the proposed method outperforms the best self-supervised method ( i.e., LRD), on almost all homogeneous graph datasets. This indicates that the proposed method is also available to learn the noise-free affinity matrix for homogeneous graphs as well as capture invariant and clustering information to benefit downstream tasks. Therefore, the effectiveness of the proposed method is verified on both heterogeneous and homogeneous graph datasets.

3.2.2 Ablation Study

The proposed method investigates the objective function J to learn the rank-constrained affinity matrix, as well as capture invariant and clustering information. To verify the effectiveness of each component of J ( i.e., L sp , L nc , and L cc ), we investigate the performance of all variants on the node classification task and report the results in Table 2.

From Table 2, we have the observations as follows. First, the proposed method with the complete objective function obtains the best performance. For example, the proposed method on average improves by 1.8%, compared to the best variant ( i.e., without L nc ), indicating that all components in the objective function are necessary for the proposed method. This is consistent with our claims, i.e., it is essential to optimize the adaptive graph structure to mitigate noisy connections as well as utilize the cluster-level information to benefit downstream tasks. Second, the variant without L sp achieves inferior results to the other two variants ( i.e., without L nc and without L cc , respectively). This can be attributed to the fact that the spectral loss L sp enforces the cluster assignment matrix to fit the eigenvectors, which is necessary for the closed-form solution of the affinity matrix.

3.2.3 Visualization

To verify the effectiveness of the learned affinity matrix and the representations for downstream tasks, we visualize the affinity matrix in the heatmap and visualize the representations with t-SNE [49] on DBLP and Aminer datasets and report the results in Figure 2.

Specifically, we randomly sample 50 nodes in each class and then visualize elements of the affinity matrix S among sampled nodes with the heatmap, where rows and columns are reordered by node labels. In the correlation map, the darker a pixel, the larger the value of the element of S . In Figures 2(a) and 2(c), the heatmaps exhibit that there are nearly c ( i.e., the number of classes) components in the affinity matrix, and almost all elements with large values fall in the block diagonal structure. This indicates that the affinity matrix indeed contains c connected components to mitigate noisy connections among different classes. Moreover, the t-SNE visualization in Figures 2(b) and 2(d) further indicate that the learned representations can be well divided into c partitions. This is consistent with the observation in Theorem 2.5 and verifies the effectiveness of the learned representations.

4 Conclusion

In this paper, we revisited previous SHGL methods from the perspective of spectral clustering and then introduced a novel framework to alleviate existing issues. Specifically, we first proved that optimizing

(a)

S

of DBLP

(b) t-SNE of DBLP

(c)

S

of Aminer

(d) t-SNE of Aminer

previous SHGL methods is equivalent to performing spectral clustering with additional regularization under the orthogonalization assumption. Then we proposed an efficient spectral clustering method with the rank constraint to learn an adaptive affinity matrix and mitigate noisy connections in previous methods. Moreover, we designed node-level and cluster-level consistency constraints to capture invariant and clustering information, thus benefiting the performance of downstream tasks. Theoretical analysis indicates that the learned representations are divided into distinct partitions based on the number of classes, and are expected to achieve better generalization ability than representations of previous SHGL methods. Comprehensive experiments verify the effectiveness of the proposed method on both homogeneous and heterogeneous graph datasets on different downstream tasks.

Potential limitations and broader impact. Our potential limitation is that this work is designed based on node features. However, in heterogeneous graphs, instances arise where nodes are devoid of features. While one-hot vectors or structural embeddings can be designated as node features to tackle this problem, we recognize the necessity of devising dedicated techniques tailored for heterogeneous graphs with missing node features. In addition, the proposed method can also be used to deal with the homophily problem, which aims to explore the connections within the same class. We consider these aspects as potential directions for future research. Despite the great development of SHGL, some theoretical foundations are still lacking. Our work theoretically connects existing SHGL methods and spectral clustering and may open a new path to understanding and designing SHGL. Besides that, we do not foresee any direct negative impacts on the society.

Acknowledgments

This project is supported by the National Key Research and Development Program of China under Grant No. 2022YFA1004100, the Natural Science Foundation of Guangdong Province of China under Grant No. 2024A1515011381, and the National Research Foundation, Singapore, under its AI Singapore Programme (AISG Award No: AISG2-RP-2021-023).

References

Liang Bai and Jiye Liang. Sparse subspace clustering with entropy-norm. In ICML , pages 561-568, 2020.

Piotr Bielak, Tomasz Kajdanowicz, and Nitesh V. Chawla. Graph barlow twins: A self-supervised representation learning framework for graphs. Knowledge-Based Systems , 256:109631, 2022.

S Boyd, L Vandenberghe, and L Faybusovich. Convex optimization. IEEE Transactions on Automatic Control , 51(11):1859-1859, 2006.

Yuxiao Dong, Nitesh V. Chawla, and Ananthram Swami. metapath2vec: Scalable representation learning for heterogeneous networks. In SIGKDD , pages 135-144, 2017.

Ky Fan. On a theorem of weyl concerning eigenvalues of linear transformations i. Proceedings of the National Academy of Sciences , 35(11):652-655, 1949.

Walter Gander. Algorithms for the qr decomposition. Res. Rep , 80(02):1251-1268, 1980.

Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. In NeurIPS , pages 5000-5011, 2021.

Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In ICML , pages 4116-4126, 2020.

Dongxiao He, Chundong Liang, Cuiying Huo, Zhiyong Feng, Di Jin, Liang Yang, and Weixiong Zhang. Analyzing heterogeneous networks with missing attributes by unsupervised contrastive learning. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Dongxiao He, Jitao Zhao, Rui Guo, Zhiyong Feng, Di Jin, Yuxiao Huang, Zhen Wang, and Weixiong Zhang. Contrastive learning meets homophily: two birds with one stone. In ICML , pages 12775-12789, 2023.

Binbin Hu, Yuan Fang, and Chuan Shi. Adversarial learning on heterogeneous information networks. In SIGKDD , pages 120-129, 2019.

Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S Yu. Leveraging meta-path based context for top-n recommendation with a neural co-attention model. In SIGKDD , pages 1531-1540, 2018.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In WWW , pages 2704-2710, 2020.

Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition confidence maximisation. In CVPR , pages 8849-8858, 2020.

Yudi Huang, Yujie Mo, Yujing Liu, Ci Nie, Guoqiu Wen, and Xiaofeng Zhu. Multiplex graph representation learning via bi-level optimization. In IJCAI , 2024.

Yiding Jiang, Parth Natekar, Manik Sharma, Sumukh K Aithal, Dhruva Kashyap, Natarajan Subramanyam, Carlos Lassance, Daniel M Roy, Gintare Karolina Dziugaite, Suriya Gunasekar, et al. Methods and analysis of the first competition in predicting generalization of deep learning. In NeurIPS , pages 170-190, 2021.

Di Jin, Luzhi Wang, Yizhen Zheng, Guojie Song, Fei Jiang, Xiang Li, Wei Lin, and Shirui Pan. Dual intent enhanced graph neural network for session-based new item recommendation. In WWW , pages 684-693, 2023.

Baoyu Jing, Chanyoung Park, and Hanghang Tong. Hdmi: High-order deep multiplex infomax. In WWW , pages 2414-2424, 2021.

P. Diederik Kingma and Lei Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

N. Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR , pages 1-14, 2017.

Marc T Law, Raquel Urtasun, and Richard S Zemel. Deep spectral clustering learning. In ICML , pages 1985-1994, 2017.

Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, Xinwang Liu, and Fuchun Sun. A survey of knowledge graph reasoning on graph types: Static, dynamic, and multimodal. arXiv preprint arXiv:2212.05767 , 2022.

Meng Liu, Ke Liang, Yawei Zhao, Wenxuan Tu, Sihang Zhou, Xinbiao Gan, Xinwang Liu, and Kunlun He. Self-supervised temporal graph learning with temporal and structural intensity alignment. IEEE Transactions on Neural Networks and Learning Systems , 2024.

Meng Liu, Yue Liu, Ke Liang, Wenxuan Tu, Siwei Wang, Sihang Zhou, and Xinwang Liu. Deep temporal graph clustering. In ICLR , 2024.

Nian Liu, Xiao Wang, Deyu Bo, Chuan Shi, and Jian Pei. Revisiting graph contrastive learning from the perspective of graph spectrum. In NeurIPS , volume 35, pages 2972-2983, 2022.

Yue Liu, Ke Liang, Jun Xia, Sihang Zhou, Xihong Yang, Xinwang Liu, and Stan Z Li. Dink-net: Neural clustering on large graphs. In ICML , 2023.

Yuanfu Lu, Chuan Shi, Linmei Hu, and Zhiyuan Liu. Relation structure-aware heterogeneous information network embedding. In AAAI , pages 4456-4463, 2019.

Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng, Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, and Jie Tang. Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks. In SIGKDD , pages 1150-1160, 2021.

Yujie Mo, Yajie Lei, Jialie Shen, Xiaoshuang Shi, Heng Tao Shen, and Xiaofeng Zhu. Disentangled multiplex graph representation learning. In ICML , pages 24983-25005, 2023.

Yujie Mo, Feiping Nie, Zheng Zhang, Ping Hu, Heng Tao Shen, Xinchao Wang, and Xiaofeng Zhu. Self-supervised heterogeneous graph learning: a homophily and heterogeneity view. In ICLR , 2024.

Yujie Mo, Zongqian Wu, Yuhuan Chen, Xiaoshuang Shi, Heng Tao Shen, and Xiaofeng Zhu. Multiplex graph representation learning via common and private information mining. In AAAI , pages 9217-9225, 2023.

Bojan Mohar, Y Alavi, G Chartrand, and OR Oellermann. The laplacian spectrum of graphs. Graph theory, combinatorics, and applications , 2(871-898):12, 1991.

Parth Natekar and Manik Sharma. Representation based complexity measures for predicting generalization in deep learning. arXiv preprint arXiv:2012.02775 , 2020.

Feiping Nie, Xiaoqian Wang, and Heng Huang. Clustering and projected clustering with adaptive neighbors. In SIGKDD , pages 977-986, 2014.

Feiping Nie, Danyang Wu, Rong Wang, and Xuelong Li. Self-weighted clustering with adaptive neighbors. IEEE Transactions on Neural Networks and Learning Systems , pages 3428-3441, 2020.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Erlin Pan and Zhao Kang. Multi-view contrastive graph clustering. In NeurIPS , volume 34, pages 2148-2159, 2021.

Chanyoung Park, Donghyun Kim, Jiawei Han, and Hwanjo Yu. Unsupervised attributed multiplex network embedding. In AAAI , pages 5371-5378, 2020.

Liang Peng, Yujie Mo, Jie Xu, Jialie Shen, Xiaoshuang Shi, Xiaoxiao Li, Heng Tao Shen, and Xiaofeng Zhu. Grlc: Graph representation learning with constraints. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In WWW , pages 259-270, 2020.

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In SIGKDD , pages 701-710, 2014.

Uri Shaham, Kelly Stanton, Henry Li, Ronen Basri, Boaz Nadler, and Yuval Kluger. Spectralnet: Spectral clustering using deep neural networks. In ICLR , 2018.

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 , 2018.

Chuan Shi, Binbin Hu, Wayne Xin Zhao, and S Yu Philip. Heterogeneous information network embedding for recommendation. IEEE Transactions on Knowledge and Data Engineering , 31(2):357-370, 2018.

Chuan Shi, Yitong Li, Jiawei Zhang, Yizhou Sun, and S Yu Philip. A survey of heterogeneous information network analysis. IEEE Transactions on Knowledge and Data Engineering , pages 17-37, 2016.

Zhiquan Tan, Yifan Zhang, Jingqin Yang, and Yang Yuan. Contrastive learning is spectral clustering on similarity graph. In ICLR , 2024.

Chang Tang, Zhenglai Li, Jun Wang, Xinwang Liu, Wei Zhang, and En Zhu. Unified onestep multi-view spectral clustering. IEEE Transactions on Knowledge and Data Engineering , 35(6):6449-6460, 2022.

Yijun Tian, Kaiwen Dong, Chunhui Zhang, Chuxu Zhang, and Nitesh V. Chawla. Heterogeneous graph masked autoencoders. In AAAI , pages 9997-10005, 2023.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(11), 2008.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR , pages 1-12, 2018.

Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In ICLR , pages 1-17, 2019.

Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing , 17:395-416, 2007.

Qi Wang, Zequn Qin, Feiping Nie, and Xuelong Li. Spectral embedded adaptive neighbors clustering. IEEE Transactions on Neural Networks and Learning Systems , pages 1265-1271, 2018.

Song Wang and Jeffrey Mark Siskind. Image segmentation with ratio cut. IEEE Transactions on Pattern Analysis and Machine Intelligence , 25(6):675-690, 2003.

Xiao Wang, Deyu Bo, Chuan Shi, Shaohua Fan, Yanfang Ye, and S Yu Philip. A survey on heterogeneous graph embedding: methods, techniques, applications and sources. IEEE Transactions on Big Data , 9(2):415-436, 2022.

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S. Yu. Heterogeneous graph attention network. In WWW , pages 2022-2032, 2019.

Xiao Wang, Nian Liu, Hui Han, and Chuan Shi. Self-supervised heterogeneous graph neural network with co-contrastive learning. In SIGKDD , pages 1726-1736, 2021.

Zehong Wang, Qi Li, Donghua Yu, Xiaolong Han, Xiao-Zhi Gao, and Shigen Shen. Heterogeneous graph contrastive multi-view learning. In SDM , pages 136-144, 2023.

Fang Wu, Dragomir Radev, and Stan Z Li. Molformer: Motif-based transformer on 3d heterogeneous molecular graphs. In AAAI , volume 37, pages 5312-5320, 2023.

Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Z Li. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering , pages 1-20, 2021.

Teng Xiao, Zhengyu Chen, Zhimeng Guo, Zeyang Zhuang, and Suhang Wang. Decoupled self-supervised learning for graphs. In NeurIPS , 2022.

Liang Yang, Cheng Chen, Weixun Li, Bingxin Niu, Junhua Gu, Chuan Wang, Dongxiao He, Yuanfang Guo, and Xiaochun Cao. Self-supervised graph neural networks via diverse and interactive message passing. In AAAI , pages 4327-4336, 2022.

Liang Yang, Runjie Shi, Qiuliang Zhang, Zhen Wang, Xiaochun Cao, Chuan Wang, et al. Self-supervised graph neural networks via low-rank decomposition. In NeurIPS , 2024.

Xihong Yang, Jin Jiaqi, Siwei Wang, Ke Liang, Yue Liu, Yi Wen, Suyuan Liu, Sihang Zhou, Xinwang Liu, and En Zhu. Dealmvc: Dual contrastive calibration for multi-view clustering. In ACM MM , pages 337-346, 2023.

Xihong Yang, Yue Liu, Sihang Zhou, Siwei Wang, Wenxuan Tu, Qun Zheng, Xinwang Liu, Liming Fang, and En Zhu. Cluster-guided contrastive graph clustering network. In AAAI , pages 10834-10842, 2023.

Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, and Wei Liu. Deep spectral clustering using dual autoencoder network. In CVPR , pages 4066-4075, 2019.

Xingtong Yu, Yuan Fang, Zemin Liu, and Xinming Zhang. Hgprompt: Bridging homogeneous and heterogeneous graphs for few-shot prompt learning. In AAAI , pages 16578-16586, 2024.

Zhaoning Yu and Hongyang Gao. Molecular representation learning via heterogeneous motif graph neural networks. In ICML , pages 25581-25594, 2022.

Rui Zhang, Arthur Zimek, and Peter Schneider-Kamp. A simple meta-path-free framework for heterogeneous network embedding. In CIKM , pages 2600-2609, 2022.

Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. Costa: Covariancepreserving feature augmentation for graph contrastive learning. In SIGKDD , pages 2524-2534, 2022.

Jianan Zhao, Xiao Wang, Chuan Shi, Binbin Hu, Guojie Song, and Yanfang Ye. Heterogeneous graph structure learning for graph neural networks. In AAAI , pages 4697-4705, 2021.

Sheng Zhou, Kang Yu, Defang Chen, Bolang Li, Yan Feng, and Chun Chen. Collaborative knowledge distillation for heterogeneous information network embedding. In WWW , pages 1631-1639, 2022.

Sihang Zhou, Xinwang Liu, Jiyuan Liu, Xifeng Guo, Yawei Zhao, En Zhu, Yongping Zhai, Jianping Yin, and Wen Gao. Multi-view spectral clustering with optimal neighborhood laplacian matrix. In AAAI , pages 6965-6972, 2020.

Yanqiao Zhu, Yichen Xu, Hejie Cui, Carl Yang, Qiang Liu, and Shu Wu. Structure-enhanced heterogeneous graph contrastive learning. In SDM , pages 82-90, 2022.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 , 2020.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW , pages 2069-2080, 2021.

A Related Work

This section briefly reviews topics related to this work, including self-supervised heterogeneous graph learning in Section A.1, and spectral clustering in Section A.2.

A.1 Self-Supervised Heterogeneous Graph Learning

In recent years, self-supervised heterogeneous graph learning (SHGL) has emerged as a helpful technique to deal with the heterogeneous graph that consists of different types of entities without needing labeled data [56, 60, 57, 55, 39, 23, 67]. As a result, SHGL captures meaningful representations of nodes and edges, enabling better performance in downstream tasks like node classification and node clustering. Due to its powerful capability, SHGL has been applied to various real applications, such as social network analysis [22, 62, 10], and recommendation systems [44, 17, 15].

Existing SHGL methods can be broadly classified into two groups, i.e., meta-path-based methods and adaptive-graph-based methods. In meta-path-based methods, several graphs are usually constructed based on different pre-defined meta-paths to examine diverse relationships among nodes that share similar labels [18, 74]. For example, STENCIL [74] and HDMI [18] construct meth-path-based graphs and then conduct node-level consistency constraints ( e.g., contrastive loss) between node representations in different graphs. In addition, HGCML [58] and CPIM [31] propose to maximize the mutual information between node representations from different meta-path-based graphs. However, pre-defined meta-paths in these methods generally require expert knowledge and prohibitive computation costs [69]. Therefore, adaptive-graph-based methods are proposed to learn the adaptive graph structures to capture the relationships among nodes that possess the same label, instead of using meta-paths. For example, recently, HERO [30] made the first attempt to learn an adaptive self-expressive matrix to capture the homophily in the heterogeneous graph, thus avoiding meta-paths.

Although existing SHGL methods (especially the adaptive-graph-based methods) have achieved impressive performance in several tasks, the learned graph structure cannot be guaranteed optimal. As a result, the learned graph structure may contain noisy connections from different classes to affect the message-passing process and weaken the discriminative information in node representations.

A.2 Spectral Clustering

Spectral clustering partitions data points into clusters based on a similarity matrix derived from the data [53, 47, 65]. Owing to its proficiency in identifying clusters with complex shapes and handling non-linearly separable data, spectral clustering is widely used in many scenarios [73, 24, 26, 64].

The spectral clustering methods can be broadly classified into two groups, i.e., traditional spectral clustering and deep spectral clustering. Traditional spectral clustering methods aim to group data points that are similar to each other while being dissimilar to points in other clusters by eigendecomposition [34, 35]. For example, CAN [34] proposes to learn the data similarity matrix and clustering structure simultaneously with the eigendecomposition. SWCAN [35] further assigns weights for different features to learn the similarity graph and partition samples into clusters simultaneously. Despite its effectiveness, traditional spectral clustering generally requires expensive computation costs, especially for large datasets. To alleviate this issue, deep spectral clustering methods have been proposed in recent years. For example, DSC [66] employs an encoder and two decoders to train the network, thus obtaining discriminative representations for clustering and implementing the cluster assignment via the neural network. DSCL [21] introduces a novel metric learning framework that leverages spectral clustering principles, thus reducing complexity to linear levels. SpectralNet [42] proposes to learn a mapping function via the orthogonalization network to address the out-of-sample-extension and scalability problems.

The above methods conduct spectral clustering explicitly. Surprisingly, recent research shows that some popular self-supervised methods also implicitly conduct spectral clustering [7, 46]. For example, [7] demonstrates contrastive learning performs spectral clustering on the population augmentation graph by replacing the standard InfoNCE [36] with its proposed spectral contrastive loss. [46] demonstrates that contrastive learning with the standard InfoNCE loss is equivalent to spectral clustering on the similarity graph. Although these methods make efforts to connect previous selfsupervised methods with spectral clustering, they cannot be easily transferred to SHGL. First, these methods are almost based on the augmentation graph, which assumes that different augmentations

of the same sample connect each other and thus form a graph. In contrast, in SHGL, there is no augmentation, and the graph is constructed by connecting different samples. Second, compared to the above methods, SHGL incorporates the message-passing process, which makes it more complex. Therefore, connecting SHGL methods with the spectral clustering remains challenging.

B Algorithm and Complexity Analysis

This section provides the pseudo-code of the proposed method in Section B.1, and the complexity analysis of our method in Section B.2.

B.1 Algorithm

Algorithm 1 The pseudo-code of the proposed method.

Input: Heterogeneous graph G = ( V , E , X , T , R ) , non-negative parameters β , γ , η , µ and δ ; Output: Encoders g ϕ , f θ ;

1: Initialize parameters;

2: while not converge do

3: Obtain semantic representations H with encoder g ϕ ;

4: Obtain the closed-form solution of the affinity matrix S by Eq. (10);

5: Obtain the orthogonal cluster assignment matrix Y by Eq. (11) and Eq. (12);

6: Conduct the spectral loss based on Y and S by Eq. (13);

7: Obtain node representations Z by Z = SH ;

8: Obtain heterogeneous representations ˜ Z with encoder f θ ;

9: Project node and heterogeneous representations into a latent space to obtain Q and ˜ Q ;

10: Conduct the node-level consistency constraint between Q and ˜ Q by Eq. (16);

11: Obtain cluster representations ˆ Q by Eq. (17);

12: Conduct the cluster-level consistency constraint between ˜ Q and ˆ Q by Eq. (18);

13: Compute the objective function J by Eq. (19);

14: Back-propagate J to update model weights;

15: end while

B.2 Complexity Analysis

Based on the Algorithm 1 above, we then analyze the time complexity of the proposed method. Recalling Eq. (10) in the main text:

where d ij = ∥ h i -h j ∥ 2 2 + β ∥ f i -f j ∥ 2 2 , where H ∈ R n × d and F ∈ R n × c are semantic representations and eigenvector matrix, d and c indicate number of dimensions and classes, and n indicates the number of nodes. To reduce the computation costs, the proposed method proposes to only calculate s ij between node v i and its k nearest neighbors. Therefore, the time complexity of Eq. (21) is O ( nk ) . Moreover, the proposed method proposes to replace the eigendecomposition with a projection head and orthogonalization layer to further reduce the time complexity. Specifically, the time complexity of the orthogonal process for H and Y with the QR decomposition is O ( nd 2 ) and O ( nc 2 ) , respectively. The time complexity of the inversion process in Eq. (12) for H and F is O ( d 3 ) and O ( c 3 ) . Moreover, the time complexity of the spectral loss is O ( nkc ) and the time complexity of Z = SH is O ( nkd ) . In addition, the time complexity of node-level and cluster-level consistency constraints are O ( nd 2 ) and O ( n ) , respectively. Therefore, the overall complexity of the proposed method is O ( nd 2 + nc 2 + nkd + nkc + d 3 + c 3 ) in each epoch, where d 2 , c 2 < n , thus is scaled linearly with the sample size.

C Proofs of Theorems

This section provides definition, detailed proofs of Theorems, and derivation process in Section 2, including the proofs of Theorem 2.2 in Section C.1, the proofs of Theorem 2.3 in Section C.2, the

proofs of Theorem 2.5 in Section C.3, the proofs of Theorem 2.6 in Section C.4, the derivation of Eq. (7) in Section C.5, the derivation of the closed-form solution and parameters in Section C.6, and the derivation of the orthogonalization in Section C.7.

C.1 Proof of Theorem 2.2

Theorem C.1. (Restating Theorem 2.2 in the main text). Assume the learned representations H are orthogonal, optimizing previous meta-path-based and adaptive-graph-based SHGL methods is equivalent to performing spectral clustering with additional regularization, i.e.,

where R ( · ) indicates the regularization term, ˆ L indicates the Laplacian matrix of the meta-path-based graph or the adaptive graph structure.

Proof. First, we prove the connection between previous meta-path-based SHGL methods and spectral clustering. To do this, take a heterogeneous graph with two meta-paths as an example, we let G = {G (1) ∪ G (2) } indicates the union of all meta-path-based graph views. Moreover, we denote the representations of previous methods before the message-passing as H (generally obtained by linear mapping from original node features). In addition, we denote the node representations of different graph views after the message-passing as Z ( r ) , respectively, where r = 1 , 2 , i.e.,

where N ( v i ) ( r ) indicates the one-hop neighbors of node v i in the r -th meta-path-based graph.

Based on the node representations Z ( r ) of each graph, previous meta-path-based SHGL methods generally propose to extract the invariant information among node representations from different meta-path-based graphs. Here, we take the Mean Squared Error (MSE) loss as a simple example to extract the invariance and then conduct an analysis of previous meta-path-based SHGL methods. Therefore, the objective function of previous meta-path-based SHGL methods can be formulated as:

Based on Eq. (23), we can rewrite Eq. (24) as:

Denote D as the degree matrix of G , denote L = D -G as the graph laplacian, and denote h i ∈ R n h j ∈ R d is the i -th column and j -th row of H , according to the spectral graph analysis in [52], we further have

Therefore, we further have

That is where Tr ( · ) indicates the matrix trace. Therefore, based on Eq. (25) and Eq. (28), we can obtain

Based on the assumption that H T H = I , we can conclude that previous meta-path-based SHGL methods, which extract the invariance among different graphs, equals the known spectral clustering with additional regularization. Note that the MSE loss in the above example can be replaced by other contrastive or non-contrastive loss ( e.g., InfoNCE [36]), and we can easily obtain similar results.

After that, we further prove the connection between recent adaptive-graph-based SHGL methods [30] and the spectral clustering. Denote the self-expressive matrix in [30] as S , and denote the representations after projection by linear transformation as H . Moreover, denote D S as the degree matrix of S and denote L S = D S -S as the graph Laplacian. Given that the self-expressive matrix is symmetrical and non-negative, the objective function of previous adaptive-graph-based SHGL methods can be formulated as:

where α and β are non-negative parameters, and d ij indicates the distance among nodes based on H or original node features. Based on the self-expressive constraint in the first term of Eq. (30), we have

̸

Therefore, for any h i where i ∈ [1 , n ] , we further have

̸

Based on the constraint s T i 1 = 1 , we obtain

Therefore, the constraint in Eq. (31) can be transformed as:

̸

In addition, we further have

̸

Similar to the proof above, we can also rewrite Eq. (30) as:

Based on the assumption that H T H = I , we can also conclude that previous adaptive-graph-based SHGL methods are equal to the known spectral clustering with additional regularization. Therefore, we complete the proof.

C.2 Proof of Theorem 2.3

To prove Theorem 2.3, we first give the definition of the graph-cut algorithm as follows.

Definition C.2. (Graph-Cut) For a given number k of subsets, the mincut approach simply consists in choosing a partition V 1 , ..., V d which minimizes

where W ( V a , V b ) := ∑ i ∈ V a ,j ∈ V b w ij indicates the weight between different subsets, and ¯ V is the complement of V .

Theorem C.3. (Restating Theorem 2.3 in the main text). Under the same assumption in Theorem 2.2, optimizing previous meta-path-based and adaptive-graph-based SHGL methods is approximate to performing the RatioCut( V 1 , . . . , V d ) algorithm that divides the learned representations into d partitions { V 1 , . . . , V d } , i.e.,

where d indicates the dimension of representations H .

Proof. Given a partition of V with n samples into d sets V 1 , ..., V d , we first define d indicator vectors h j = ( h 1 ,j , ..., h n,j ) ′ by

Then we set the matrix H ∈ R n × d as the matrix containing those d indicator vectors as columns. Observe that the columns in H are orthonormal to each other, i.e., H T H = I , where I is the identity

̸

matrix. Denote L as the unnormalized graph Laplacian, according to [52], we can obtain

Moreover, we have h T i Lh i = ( H T LH ) ii . Therefore, we have

where Tr( · ) indicates the trace of a matrix. Therefore, minimizing the RatioCut( V 1 , . . . , V k ) can be transferred to

Then we consider relaxing the constraints of the problem by allowing the entries of the matrix H to assume arbitrary real values. As a result, the problem is transformed into a relaxed version:

This is the standard spectral clustering, as we mentioned above. Therefore, we can obtain that conducting spectral clustering is approximating to conducting the RatioCut algorithm. which divide the learned representations into d partitions, where d indicates the dimension of representations. Thus, we complete the proof.

C.3 Proof of Theorem 2.5

Theorem C.4. (Restating Theorem 2.5 in the main text). Optimizing the spectral loss L sp leads to performing the spectral clustering based on the affinity matrix S with c connected components and conducting RatioCut ( V 1 , . . . , V c ) algorithm to divide the learned representations into c partitions, i.e.,

Proof. According to Ky Fan's Theorem [5], the spectral loss L sp can be written as:

Therefore, optimizing the proposed method with L sp is equivalent to performing the spectral clustering with additional regularization.

Note that, under the orthogonal constraint, the minimum of L sp is attained when the column space of Y is the subspace of the c eigenvectors corresponding to the smallest c eigenvalues of L S . In other

words, the learned Y can perfectly fit the eigenvectors when the minimum of L sp is attained. Recall the objective function in the main text, i.e.,

Therefore, when the minimum of L sp is attained, the constraints in the above function can be satisfied, i.e., rank( L S ) = n -c holds. As a result, we can obtain the affinity matrix S with exactly c connected components.

Moreover, according to the Theorem C.3, we have

That is, the proposed method divides the learned representations into c partitions, where c indicates the number of classes. Thus, we complete the proof.

C.4 Proof of Theorem 2.6

We first follow previous works [33] to define the Complexity Measure to evaluate the generalization ability of neural networks based on the Davies Bouldin Index.

Definition C.5. (Complexity Measure) The complexity measure of neural networks can be defined as:

̸

where i and j are indices of two different classes, O ( τ ) i is the output representation of the τ -th sample belonging to class i for the given model, µ i is the cluster centroid of the representations of class i , S i is a measure of scatter within representations of class i , and M i,j is a measure of separation between representations of classes i and j .

Moreover, we further follow previous works [33, 16] to define the generalization bound G of a model based on the model complexity, i.e.,

Definition C.6. (Generalization Bound) For any δ ∈ [0 , 1] , with probability at least 1 -δ , the generalization bound G of a model follows the inequality, i.e.,

where ( x i , y i ) is a pair of labeled data, f is the model, l is the loss function, n is the number of labeled data, C is the model complexity measure.

Based on the Definitions above, we can derive the Theorem as follows.

Theorem C.7. (Restating Theorem 2.6 in the main text). The proposed method with dual consistency constraints achieves a lower boundary of the model complexity C and a higher generalization ability boundary G than previous SHGL with the node-level consistency constraint only, i.e.,

where inf( · ) and sup( · ) indicates lower bound and upper bound, respectively.

Proof. We take the binary classification as an example, Eq. (48) can be rewritten as S 0 + S 1 M 0 , 1 . Then, for the heterogeneous representations ˜ Z learned by the heterogeneous encoder, we can obtain its cluster

centroid µ 0 :

k Therefore, we can obtain

where N ( v i ) indicates the neighbors of v i from other type of nodes, W indicates the parameters of the heterogeneous encoder, µ X 0 indicates the cluster centroid of the node features of class i , µ X j indicates the cluster centroid of the node features of other types of nodes. Similarly, we further have:

where µ X indicates the cluster centroid of the node features of other types of nodes.

Moreover, we have

where < · , · > is inner production. To rewrite the above function, we first derive the following inequality, i.e.,

where 0 ≤ a ≤ 1 , and 0 ≤ b, c . To prove the inequaility in Eq. (56), we construct function f ( a ) = a 2 b +(1 -a ) 2 c -bc b + c . We then take the derivative of f ( a ) , i.e.,

Then we let f ′ ( a ) = 0 , and have a = c b + c . We have

In addition, we take the second-order derivative of f ( a ) and obtain

Therefore, f ( a ) is decreasing when a < c b + c and increasing when a > c b + c , and reaches its minimum 0 at c b + c . As a result, f ( a ) ≥ 0 always holds for 0 ≤ a ≤ 1 . Thus, we prove the inequality in Eq. (56).

Given the above inequality in Eq. (56), for Eq. (55), we let σ 2 0 = E [ ∥ W ( X ( i ) 0 -µ X 0 ) ∥ 2 ] , σ 2 1 = E [ ∥ W ( X ( i ) 1 -µ X 1 ) ∥ 2 ] , σ 2 j = E [ ∥ W ( X ( i ) j -µ X j ) ∥ 2 ] , and σ 2 k = E [ ∥ W ( X ( i ) k -µ X k ) ∥ 2 ] . Moreover, we replace a , b , and c in Eq. (56) with P 0 , σ 2 0 , and σ 2 j , respectively. Then Eq. (55) can be rewritten as:

Similarly, we can also reach the following inequality:

Therefore, the complexity measure C can calculated by:

Note that the cluster-level consistency constraint minimizes the first term in the S 2 0 and S 2 1 ( i.e., σ 2 0 and σ 2 1 ). Moreover, we can observe that Eq. (60) and Eq. (61) are the increasing function with respect to σ 0 and σ 1 . Therefore, minimizing the cluster-level consistency constraint is equivalent to minimizing the lower bound of the model complexity. Therefore, the lower bound of complexity measure C of the model with the dual consistent constraints is less than the model without it, i.e., inf( C SCHOOL ) < inf( C SHGL ) . As a result, according to [33], we can conclude that the representations learned by the dual consistent constraints have a higher bound of generalization ability than previous methods with instance-level constraint only, i.e., sup( G SCHOOL ) > sup( G SHGL ) thus we complete the proof.

C.5 Derivation of Eq. (7).

Recalling Eq. (7), i.e.,

where F ∈ R n × c is the eigenvector ( i.e., F T F = I ) of L S corresponding to the c eigenvalues. We first derive the first equation. The eigendecomposition of the symmetric L S can be written as: L S = B Λ B T , where B is the eigenvector matrix and Λ is the diagonal matrix whose diagonal elements are the eigenvalues of L S . We have:

where ∑ c i =1 Λ i indicates the sum of any c eigenvalues of L S . Obviously, min F T F = I Tr( F T L S F ) achieves its minimization when F is the eigenvectors corresponding to c smallest eigenvalues. Therefore we have min S ∑ c i =1 τ i ( L S ) = min F T F = I Tr( F T L S F ) and the first equation in Eq. (63) is proved. Moreover, based on Eq. (26)-Eq. (28), we can further complete the proof the second equation in Eq. (63).

C.6 Derivation of the Closed-Form Solution and Parameters.

We first obtain the Lagrangian function of the objective function in Eq. (9) in the main text:

where λ and ε i ≥ 0 are the Lagrangian multipliers. Based on the KKT condition [3], we can obtain the closed-form solution of the above Lagrangian function, i.e.,

where ( · ) + indicates max {· , 0 } . For the sparse affinity matrix S , each vector s i contains k nonzero elements only. Therefore, we have s ik ≥ 0 and s i,k +1 = 0 . That is, -1 2 α i d ik + λ i > 0 and -1 2 α i d i,k +1 + λ i ≤ 0 . Then based on Eq. (66) and the constraint s T i 1 = 1 , we further have

We further have

Therefore, we obtain λ i = 1 k + 1 2 kα i ∑ k j =1 d ij . Moreover, we have the following inequality for α i , i.e.,

Hence, to achieve an optimal solution s i contain precisely k non-zero values, we can set α i as:

Then the overall α could be set to the mean of α 1 , α 2 , ..., α n . Then α can be obtained by:

C.7 Derivation of the Orthogonalization

Recalling Eq. (12) in the main text:

where P is the cluster assignment matrix, and R is a upper triangular matrix obtained from the QR decomposition P = ER and E T E = I . Then we have

Therefore, we can obtain that Y is orthogonal.

D Experimental Settings

This section provides detailed experimental settings in Section Experiments, including the description of all datasets in Section D.1, summarization of all comparison methods in Section D.2, evaluation protocol in Section D.3, model architectures and settings in Section D.4, and computing resource details in Section D.5.

D.1 Datasets

We use four public heterogeneous graph datasets and two public homogeneous graph datasets from various domains. Heterogeneous graph datasets include three academic datasets ( i.e., ACM [56], DBLP [56], and Aminer [11]), and one business dataset ( i.e., Yelp [71]). Homogeneous graph datasets include two sale datasets ( i.e., Photo and Computers [43]). Table 3 summarizes the data statistics. We list the details of the datasets as follows.

ACM is an academic heterogeneous graph dataset. It contains three types of nodes (paper (P), author (A), subject (S)), four types of edges (PA, AP, PS, SP), and categories of papers as labels.

Yelp is a business heterogeneous graph dataset. It contains four types of nodes (business (B), user (U), service (S), level (L)), six types of edges (BU, UB, BS, SB, BL, LB), and categories of businesses as labels.

DBLP is an academic heterogeneous graph dataset. It contains three types of nodes (paper (P), authors (A), conference (C)), four types of edges (PA, AP, PC, CP), and research areas of authors as labels.

Aminer is an academic heterogeneous graph dataset. It contains three types of nodes (paper (P), author (A), reference (R)), four types of edges (PA, AP, PR, RP), and categories of papers as labels.

Photo and Computers are two co-purchase homogeneous graph datasets. They are two networks extracted from Amazon's co-purchase data. Nodes are products, and edges denote that these products were often bought together. Products are categorized into several classes by the product category.

D.2 Comparison Methods

The comparison methods include eleven heterogeneous graph methods and twelve homogeneous graph methods. Heterogeneous graph methods include Mp2vec [4], HAN [56], HGT [13], DMGI [38], DMGIattn [38], HDMI [18], HeCo [57], HGCML [58], CPIM [31], HGMAE [48], and HERO [30]. Homogeneous graph methods include GCN [20], GAT [50], DeepWalk [41], DGI [51], GMI [40], MVGRL [8], GRACE [75], GCA [76], G-BT [2], COSTA [70], DSSL [61], and LRD [63]. The characteristics of all methods are listed in Table 4, where 'Hetero' and 'Homo' indicate the methods designed for the heterogeneous graph and homogeneous graph, respectively. 'Semisup", and 'Self-sup/unsup" indicate that the method conducts semi-supervised learning, and selfsupervised/unsupervised learning, respectively. 'Meta-path' indicates that the method requires pre-defined meta-paths during the training process. 'Adaptive' indicates that the method learns an adaptive graph structure instead of traditional meta-paths.

D.3 Evaluation Protocol

We follow the evaluation in previous works [18, 37, 72] to conduct node classification and node clustering as semi-supervised and unsupervised downstream tasks, respectively. Specifically, we first train models with unlabeled data in a self-supervised manner and output learned node representations. After that, the resulting representations can be used for different downstream tasks. For the node classification task, we train a simple logistic regression classifier with a fixed iteration number, and then evaluate the effectiveness of all methods with Micro-F1 and Macro-F1 scores. For the node clustering task, we conduct clustering and split the learned representations into c clusters with the K-means algorithm, then calculate the normalized mutual information (NMI) and average rand index (ARI) to evaluate the performance of node clustering.

D.4 Model Architectures and Settings

As described in Section 2, the proposed method employs the MLP ( i.e., g ϕ ) and the closed-form solution of the affinity matrix S to obtain node representations Z . Moreover, the proposed method employs the heterogeneous encoder ( i.e., f θ ) to obtain heterogeneous representations ˜ Z . In addition, the proposed method employs the projection head p φ to obtain the cluster assignment matrix P . After that, the proposed method employs projection head q γ to map the node representations and heterogeneous representations into latent spaces. In the proposed method, projection head p φ and q γ are simply implemented by the linear layer, followed by the ReLU activation. We report the settings for the dimensions of encoders in Table 5. Finally, In the proposed method, all parameters were optimized by the Adam optimizer [19] with an initial learning rate. Moreover, We use early stopping with a patience of 30 to train the proposed SHGL model. In all experiments, we repeat the experiments five times for all methods and report the average results.

D.5 Computing Resource Details

All experiments were implemented in PyTorch and conducted on a server with 8 NVIDIA GeForce 3090 (24GB memory each). Almost every experiment can be done on an individual 3090, and the training time of all comparison methods as well as our method, is less than 1 hour.

(a) HDMI (SIL: 0.36)

(b) HeCo (SIL: 0.33)

(c) HERO (SIL: 0.34)

(d) SCHOOL (SIL: 0.41)

E Additional Experiments

This section provides some additional experimental results to support the proposed method, including experiments on the effectiveness of the affinity matrix in Section E.1, visualization of the learned representations in Section E.4, parameter analysis in Section E.5, experimental results on the node clustering task in Table 6, and experimental results on homogeneous graph datasets in Table 7.

E.1 Effectiveness of the Rank-Constrained Affinity Matrix

The proposed method proposes to learn a rank-constrained affinity matrix with exact c components to capture the connections within the same class while mitigating the connections from different classes. This actually shares part of a similar idea with the self-attention mechanism, which aims to assign weights for all sample pairs. To further verify the effectiveness of the rank-constrained affinity matrix, we investigate the performance of the variants methods with the cosine similarity, the affinity matrix, the self-attention mechanism, and report the results in Table 8.

Obviously, the proposed method with the affinity matrix obtains superior performance than the cosine similarity and the self-attention mechanism on all datasets. The reason can be attributed to the fact that the affinity matrix in the proposed method is constrained to contain exactly c components to mitigate noisy connections from different classes. In contrast, although either the cosine similarity or self-attention mechanisms may assign small weights for node pairs from different classes, it inevitably introduces noise during the message-passing process to affect the quality of node representations. As a result, the effectiveness of the rank-constrained affinity matrix is verified.

E.2 Effectiveness of the Node-level Consistency Constraint

To verify the effectiveness of the node-level consistency constraint, we conducted experiments to replace the proposed node-level consistency constraint with the InfoNCE loss and reported the

results in Table 9. From Table 9, we can find that the variant method with InfoNCE loss obtains a similar performance to the proposed method. However, the InfoNCE loss generally requires the time complexity of O ( n 2 ) , where n is the number of nodes. This may introduce large computation costs during the training process. In contrast, the proposed method simply designs the node-level consistency constraint in Eq. (15) to capture the invariant information with the time complexity of O ( nd 2 ) , where d is the representation dimension and generally d 2 < n .



c



































E.3 Effectiveness of Different Cluster Numbers

The proposed method divides the learned representations into several clusters. Generally, the number of clusters equals to c obtains better results because, in downstream tasks, it is easier to distinguish c clusters than a larger number of clusters. To verify it, we changed the number of clusters and reported the results in Figure 4. Obviously, the proposed method obtains the best results when the number of clusters equals to c and decreases as the number of classes increases. This is reasonable because when the number of classes increases, the nodes within the same class may be assigned to different clusters, thus making it difficult to classify them correctly.

E.4 Visualization of the Learned Representations

To further verify the effectiveness of the learned representations, we visualize node representations of the proposed SCHOOL and other SHGL comparison methods on the DBLP dataset and report the results and corresponding silhouette scores (SIL) in Figure 3. Obviously, in the visualization,

δ

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 3

μ

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 3

Macro-F1 (%)

80

82

84

86

88

90

92

94

(a) ACM

δ

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 3

μ

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 3

Macro-F1 (%)

90

91

92

93

94

95

(b) Yelp

δ

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 3

μ

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 3

Macro-F1 (%)

90

91

92

93

94

95

(c) DBLP

δ

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 3

μ

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 3

Macro-F1 (%)

50

55

60

65

70

75

80

(d) Aminer

the node representations learned by the proposed method exhibit better clustering status, i.e., nodes with different class labels are more widely separated. Moreover, the representations learned by the proposed method obtain the best silhouette score, compared to other SHGL comparison methods ( i.e., HDMI, HeCo, and HERO). The reason can be attributed to the fact that the proposed method conducts spectral clustering explicitly, and cuts the learned graph into c components as well as further utilizes the clustering information to facilitate downstream tasks.

E.5 Parameter Analysis

In the proposed method, we employ non-negative parameters ( i.e., µ , and δ ) to achieve a trade-off between different terms of the final objective function J . To investigate the impact of µ , and δ with different settings, we conduct the node classification on all heterogeneous graph datasets by varying the value of parameters in the range of [10 -3 , 10 3 ] and reporting the results in Figure 5. From Figure 5, we can find that if the values of parameters are too small ( e.g., 10 -3 ), the proposed method cannot achieve satisfactory performance. This verifies that both node-level and cluster-level consistency constraints are significant for the proposed method.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] We discuss the limitations of the work in Section 4.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] We provide the assumptions and complete proof in Appendix C.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] See Section 3 and Appendix D.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] We released codes and data at https://github.com/YujieMo/SCHOOL .

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] We specify all the training and test details in Appendix D.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] We list the details of experiments compute resources in Appendix D.5.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes] We discuss broder impacts in Section 4.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework that revisits SHGL from a spectral clustering perspective, incorporating rank and dual consistency constraints. This approach uses a rank-constrained spectral clustering method to refine the affinity matrix and remove noise, while also integrating node-level and cluster-level consistency constraints to better capture and use invariant and clustering information. Theoretical analysis and experimental results show that this new method significantly improves performance in downstream tasks compared to existing methods.\n\nSTRENGTHS:\n1. This paper provides a theoretical investigation of previous SHGL methods from the perspective of spectral clustering.\n2. This paper proposes a new framework to capture the cluster-level graph invariant representations.\n3. Experiments demonstrate the effectiveness of the proposed methods.\n\nWEAKNESSES:\nUnfortunately, this paper is not easy to follow due to the missing parts in terms of motivation and logic.\n\n1. In the introduction part, the authors mention three challenges abruptly from Lines 44 to 48. However, it is unclear why they are challenging to address. For example, why it is difficult to understand previous SHGL methods from the clustering perspective?  It is also unclear the logical relationships between the three challenges.\n\n2. In Section 2, the authors introduce some notations about the heterogeneous graph. However, it is unclear what is the objective of the problem.\n\n3. In Section 2.1, it is unclear why authors need to bridge the gap between SHGL methods and graph-cut algorithm, and what is its benefits.\n\n4.  In Section 2.2, it is unclear why the rank constraint can mitigate noisy connections.\n\n5. When reading Section 2.2, the reviewer gets lost in the lemma and equation details. It is better if the authors can highlight the contributions and leave all technical derivations to the appendix.\n\n6. The logical relationship between Sections 2.2 and 2.3 is unclear. Why both the low-rank constraints and dual consistency constraints should be imposed, and what is the relationship between these two constraints?\n\n7.  In the experiments, it is unclear why two homogenous graph datasets were used for performance evaluation, given the objective of this paper is for heterogeneous graph learning.\n\nQUESTIONS:\nPlease find questions in the weaknesses above.\n\nLIMITATIONS:\nThis paper does not mention the limitations of this work or the potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work deals with the problem of self-supervised representation learning in heterogeneous graphs. First, a spectral-clustering based objective is presented to unify the objectives of existing methods. Second, a novel self-supervised method is proposed that tries to capture both the cluster information and node-level information in the learnt representation. Experiments show the effectiveness of the proposed method over existing methods.\n\nSTRENGTHS:\n1. The proposed method is novel and follows a principled design.\n2. Presented theoretical analysis is helpful in deeper understanding of existing and proposed method.\n3. Experiemental section is comprehensive with adequate baselines, datasets, and ablations.\n\nWEAKNESSES:\n1. Presented theorems are often unclear or imprecise. For example\n\n   (a) In Theorem 2.6, authors use the term model complexity abruptly without any definition or reference. I could not understand the relation between the infimum and supremum of model complexity with generalization ability (which also needs to be rigorously defined).\n\n   (b) In Theorem 2.2, what exactly are the previous meta-path-based and adaptive-graph-based SHGL method mentioned in the theorem statement? What is the expression for the regularization term for the corresponding previous methods?\n\n   (c) There does not seem to be a clear delineation between what already exists in the literature and what is the new contribution of the paper. E.g., does Theorem 2.3 directly follow from existing results in the literature, or does this require custom analysis?\n\n   (d) Theorem 2.5 is not precise enough. When referring to \"proposed method\", it would be more precise to refer to specific objective function or algorithm block. Also \"equivalent to performing spectral clustering based on the affinity matrix ....\". Authors are recommended to replace the language description with precise objective function.\n\n\n\n2. The writing is often unclear and could be greatly improved. Several terminologies (which might not be standard in the literature) are never explained/defined. For example:\n\n   (a) Section 1 Line 42 \".. it is feasible to analyse ..\", it is not clear what exactly is feasible to analyse (although after Section 2.1, it can be understood).\n\n   (b) In abstract and introduction, it is not clear what is invariant information is\n\n   (c) Sentence in line 216-218 is unclear.\n\n   (d) How does H^T H = I imply statistical independence?\n\nQUESTIONS:\nPlease refer to the questions and comments in Weaknesses section.\n\nLIMITATIONS:\nLimitations and impacts are adequately discussed in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a theory-backed method for Self-Supervised Heterogeneous Graph Learning (SHGL) based on spectral clustering and incorporates rank constraint and node/cluster consistency regularizers to generate better embeddings. In specific, the authors start by showing that existing algorithms divide the representations into a certain number of clusters that are much larger than the number of real classes. Then, an objective function with rank constraint is proposed to reduce the noise of message-passing and consistency terms are employed to improve downstream task performance. Experiments on the real datasets show that the proposed method outperforms other baselines.\n\nSTRENGTHS:\n1.\tThe paper analyzes the issue of the exiting methods that are based on meta-path-based graph or the adaptive graph structure theoretically. Furthermore, a rank and consistency constrained approach is proposed to tackle know issues of SHGL and it is shown to have less complexity theoretically.\n2.\tExperimental results show the superiority of the proposed method consistently over a variety of datasets.\n\nWEAKNESSES:\n1.\tNotations are not always clearly defined. For example, what are the dimensions of the mapping $g_{\\phi}$ and $p_{\\phi}$?\n2.\tThe derivation of the first loss term is not clear to me. Why is there an entropy constraint term in (13)? How do we translate Eq. (8) to Eq. (13) and what is the correspondence of the terms? Line 183 states that ‘fitting eigenvectors F by (13)’, but it seems that F does not appear (13). Or is it that (13) is only for solving the last term in (8)? Clarity can be further improved upon the objective function, especially the term $\\cal{L}_{sp}$.\n3.\tWhy is $Y$ a probability matrix? Is there any guarantee on that? Only orthogonality is proved for the matrix. Do we need a simplex constraint on the columns of $Y$?\n\nQUESTIONS:\n1.\tIn Line 172, whose independence are we referring to? Why $H^T H=I$ is equivalent to statistical independence?\n2.\tIn practice, is it always the case that number of clusters equal to $c$ will give the optimal results?\n3.\tHow do we reach the inequality in (53) and (54)? Does the term in (55) correspond to the loss function in (17)?\n\nLIMITATIONS:\nThe authors are encouraged to discuss the limitations in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOverall, this paper makes the first attempt to theoretically revisit previous SHGL methods from the spectral clustering perspective in a unified manner. Specifically, this paper revisits SHGL from the spectral clustering and introducing a novel framework enhanced by rank and dual consistency constraints. Specifically, the proposed framework incorporates a rank-constrained spectral clustering method that refines the affinity matrix to exclude noise effectively. Additionally, the proposed method integrates node-level and cluster-level consistency constraints that concurrently capture invariant and clustering information to facilitate learning in downstream tasks.\n\nSTRENGTHS:\n1. I really appreciate the idea that revisiting previous SHGL methos from the perspective of spectral clustering, which is interesting and may inspire some researchers in the graph learning as well as the self-supervised learning communities.\n2. Theoretical analysis verifies the effectiveness of the proposed method, which divides the learned representations into distinct partitions based on the number of classes. Furthermore, the proposed method exhibits enhanced generalization ability than previous SHGL methods.\n3. Extensive experiments on both heterogeneous and homogeneous graphs demonstrate the effectiveness of the proposed method. Visualization and case studies further verify the claims in this paper.\n4. The proposed rank-constrained spectral clustering is novel and interesting, and it seems like can also be used to the graph structure learning in the homogeneous graph.\n\nWEAKNESSES:\n1. In the Introduction, the authors claim that previous methods conduct message-passing relying on meta-path-based graphs and adaptive graph structures, which inevitably include noise. Are there any real examples to better illustrate the noise in meta-path-path based graph as well as the adaptive graph structures?\n2. In the dual consistency constraints, the proposed method designs the node-level consistency constraint to capture the invariant information between the node representations and the heterogeneous representations. How about replacing the node-level consistency constraint with other common loss, such as InfoNCE [1]?\n[1] Aaron van den Oord et al., Representation learning with contrastive predictive coding.\n3. The authors design the rank-constrained spectral clustering to learn the affinity matrix for nodes belonging to the same node type. Although the visualization verifies the effectiveness of the affinity matrix, it would be better for the authors to add some ablations studies to further verify it, such as replacing the affinity matrix with a self-attention mechanism or simple cosine similarity.\n4. What are the specific processes for different downstream tasks (e.g., node classification and node clustering)? It seems like training the proposed method first, and then freezing parameters of the model and applying outputted representations for downstream tasks.\n5. The proposed method is designed for the heterogeneous graph while it can be implemented on both the homogeneous graph datasets and heterogeneous graph datasets. Is it just to replace the heterogeneous graph encoder with the graph convolutional layer? How to generate two different views for the dual consistency constraints?\n6. It would be better to add some recent works about the self-supervised heterogeneous graph learning in the related work, especially those published in the past two years.\n7. The paper needs further proofreading. For example, \n- In Eq. (17) I know $q_i$ is the i-th projected representation. What’s the $\\hat{\\mathbf{q}}_{\\mathbf{y}_i}$ actually means?\n- The definitions of some symbols need to be further determined, such as \\mathbf{I} in Definition 2.1. I guess it might represent the identity matrix. \n- Some mistakes in Table 3, the Freebase dataset should be removed from the Table.\n- The caption of Table 7 should be fixed.\n\nQUESTIONS:\nsee weakness.\n\nLIMITATIONS:\nYes, the authors have discussed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}