Title: Self-supervised Graph Neural Networks via Low-Rank Decomposition
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph neural network

Self-supervised learning

Low-Rank recovery
Abstract: Self-supervised learning is introduced to train graph neural networks (GNNs) by employing propagation-based GNNs designed for semi-supervised learning tasks. Unfortunately, this common choice tends to cause two serious issues. Firstly, global parameters cause the model lack the ability to capture the local property. Secondly, it is difficult to handle networks beyond homophily without label information.
This paper tends to break through the common choice of employing propagation-based GNNs, which aggregate representations of nodes belonging to different classes and tend to lose discriminative information. If the propagation in each ego-network is just between the nodes from the same class, the obtained representation matrix should follow the low-rank characteristic. To meet this requirement, this paper proposes the Low-Rank Decomposition-based GNNs (LRD-GNN-Matrix) by employing Low-Rank Decomposition to the attribute matrix. 
Furthermore, to incorporate long-distance information, Low-Rank Tensor Decomposition-based GNN (LRD-GNN-Tensor) is proposed by constructing the node attribute tensor from selected similar ego-networks and performing Low-Rank Tensor Decomposition. The employed tensor nuclear norm facilitates the capture of the long-distance relationship between original and selected similar ego-networks. Extensive experiments demonstrate the superior performance and the robustness  of  LRD-GNNs.

Self-supervised Graph Neural Networks via Low-Rank Decomposition

Abstract

Self-supervised learning is introduced to train graph neural networks (GNNs) by employing propagation-based GNNs designed for semi-supervised learning tasks. Unfortunately, this common choice tends to cause two serious issues. Firstly, global parameters cause the model lack the ability to capture the local property. Secondly, it is difficult to handle networks beyond homophily without label information. This paper tends to break through the common choice of employing propagation-based GNNs, which aggregate representations of nodes belonging to different classes and tend to lose discriminative information. If the propagation in each ego-network is just between the nodes from the same class, the obtained representation matrix should follow the low-rank characteristic. To meet this requirement, this paper proposes the Low-Rank Decomposition-based GNNs (LRD-GNN-Matrix) by employing Low-Rank Decomposition to the attribute matrix. Furthermore, to incorporate long-distance information, Low-Rank Tensor Decomposition-based GNN (LRD-GNN-Tensor) is proposed by constructing the node attribute tensor from selected similar ego-networks and performing Low-Rank Tensor Decomposition. The employed tensor nuclear norm facilitates the capture of the long-distance relationship between original and selected similar ego-networks. Extensive experiments demonstrate the superior performance and the robustness of LRD-GNNs.

1 Introduction

Graph Neural Networks (GNNs) have been successfully employed by many fields including computer vision (CV), natural language processing (NLP), information retrieval (IR), etc [1, 2, 3]. Vanilla GNNs, such as Graph Convolutional Network (GCN) [4] and Graph Attention Network (GAT) [5], are designed for semi-supervised node classification task, where part of nodes are labeled. To make models adaptive to networks beyond hompohily, flexible GNNs with more trainable parameters are proposed, such as JKNet [6], GCNII [7], H2GCN [8], FAGCN [9] and GPRGNN [10]. The role of label information is more critical for these flexible GNNs.

Inspired by the superior performance in CV and NLP, self-supervised learning, especially contrastive learning, is introduced to training GNNs. While a large number of methods are proposed, they can

be divided into two categories, i.e., contrastive and predictive models. The predictive models, such as Graph AutoEncoder (GAE) and its variational version (VAGE), MGAE [11], GALA [12] and GROVER, require data-label pairs, where the labels are self-generated from the data and significantly different for different tasks. Without the sensitive generated labels, contrastive models, which only require data-data pairs, attract more attention. The representative graph contrastive learning methods include DGI [13], GMI [14], MVGRL [15], GRACE [16], GCA [17] and BGRL [18].

Most existing graph self-supervised learning models employ vanilla GNNs designed for semisupervised learning with global parameters as encoders. Especially, GCN with one or two layers, where only the feature mappings are trainable, is the most familiar choice. Unfortunately, this common choice tends to cause two serious issues. Firstly, global parameters cause the model lack the ability to capture the local property. For example, the mapping functions in GCN are shared by all nodes. Without the supervision from labels, the reliability of global parameters is weaker than that of the local ones. Secondly, it is difficult to handle networks beyond homophily. Most universal GNNs for networks beyond homophily introduce additional trainable parameters to distinguish neighbourhoods from different classes with the help of labels, such as GPRGNN and FAGNN. Without the label information, this process becomes complicated and fallible.

This paper tends to break through the common choice of employing propagation-based GNNs in self-supervised graph learning. Propagation-based GNNs seek node representation by averaging the representations of nodes, which may belong to different classes, from neighbourhoods, and thus tends to lose discriminative information. To overcome this issue, an intuitive and ideal way is to make the propagation in each ego-network just between the nodes from the same class. By this way, the obtained representation matrix for ego-network should follow the low-rank characteristic. To meet this requirement, this paper proposes the Low-Rank Decomposition-based GNNs (LRD-GNN) by employing Low-Rank Decomposition to the attribute matrix. By directly applying Low-Rank Matrix Decomposition to the attribute matrix of the ego-network, Low-Rank Matrix Decomposition-based GNN (LRD-GNN-Matrix) is given.

Furthermore, to incorporate long-distance information, Low-Rank Tensor Decomposition-based GNN (LRD-GNN-Tensor) is proposed by extending LRD-GNN-Matrix to tensor version. LRDGNN-Tensor constructs the node attribute tensor by selecting similar ego-networks and splicing the attribute matrices of similar ego-networks into 3-way tensor. Motivated by the Tensor RPCA, tensor nuclear norm is defined as the average of the nuclear norm of all the frontal slices of discrete Fourier transformation of tensor along the 3-rd dimension. This tensor nuclear norm is equivalent to the matrix nuclear norm of the block circulant matrix of the tensor, which facilitates the capture of the long-distance relationship between original ego-network and selected similar ego-networks.

The main contributions of this paper are summarized as follows:

We point out the necessity of designing specific encoder for self-supervised graph learning.

We theoretically analyze the low-rank property of the representation matrix in ego-network.

We proposed Low-Rank Decomposition-based GNNs (LRD-GNN) and its two instances, LRD-GNN-Matrix and LRD-GNN-Tensor.

We experimentally evaluate the superior performance and the robustness of LRD-GNN.

2 Notations and Preliminaries

Notations: In this paper, boldface Euler script letters, e.g. , A , are utilized to denote tensors. Matrices are denoted by boldface capital letters, e.g. , A ; vectors are denoted by boldface lowercase letters, e.g. , a , and scalars are denoted by lowercase letters, e.g. , a . For a 3-way tensor A , its ( i, j, k ) -th entry is represented as A ijk or a ijk and use A ( i, : , :) , A (: , i, :) and A (: , : , i ) to denote respectively the i -th horizontal, lateral and frontal slice. The frontal slice A (: , : , i ) is denoted compactly as A ( i ) .

Let G = ( V , E ) denote a graph with node set V = { v 1 , v 2 , · · · , v N } and edge set E , where N is the number of nodes. The topology of graph G can be represented by its adjacency matrix A = [ a ij ] ∈ { 0 , 1 } N × N , where a ij = 1 if and only if there exists an edge e ij = ( v i , v j ) between nodes v i and v j . The degree matrix D is a diagonal matrix with diagonal element d i = ∑ N i =1 a ij as the degree of node v i . N ( v i ) = { v j | ( v i , v j ) ∈ E} stands for the neighbourhoods of node v i . Let G i = ( V i , E i ) represents the ego-network around node v i , where V i = N ( v i ) ∪ v i and E i denotes

1

2

3

4

5

6

7

Low-Rank Node

Representation

=

Weighted

Average

(b) Existing Propagation-based GNNs

(a) Ego-network

Node

Representation

=

Node

Attribute

Affinity

Matrix

X

=

+

Node

Attribute

Noise

Low-Rank Node

Representation

(c) Low-Rank

Matrix

Decomposition-based GNN

(d) Low-Rank

Tensor

Decomposition-based GNN

=

+

Node

Attribute

Noise

Low-Rank

Node

Representation

Feature Matrices Extraction

Low-Rank

Tensor

Decomposition

1

2

3

0

0

1

2

3

edges between nodes in V i . X ∈ R N × F and H ∈ R N × F ′ denote the collections of node attributes and representations with the i th rows, i.e., x i ∈ R F and h i ∈ R F ′ , corresponding to node v i , where F and F ′ stand for the dimensions of attribute and representation. For convenience, X i ∈ R ( d i +1) × F and H i ∈ R ( d i +1) × F ′ denote collections of attributes and representations of ego-network around v i .

Graph Neural Networks: Most of the Graph Neural Networks (GNNs) follow an aggregationcombination strategy [3], where each node representation is iteratively updated by aggregating node representations in the local neighbourhoods and combining the aggregated representations with the node representation itself as

where ¯ h k v stands for the aggregated representation from local neighbourhoods. Besides of the concatenation based implementation, such as GraphSAGE [19] and H2GCN [8], averaging (or summation) has been widely adopted to implement COMBINATE k ( · , · ) , such as GCN [4], GAT [5], GIN [20], etc. Except for the MAX and LSTM implementations in GraphSAGE [19], most of the GNNs utilize averaging function to implement AGGREGATE k . Therefore, they can be unified as

where W k represents the learnable parameters and σ ( · ) denotes the nonlinear mapping function.

3 Low-Rank Decomposition-based GNN

3.1 Motivations

Existing propagation-based GNNs tend to lose discriminative information. As shown in Figure 1(b) and Eq. (2), propagation-based GNNs seek node representation by averaging the representations of nodes from neighbourhoods. Unfortunately, neighbourhoods may belong to different classes from the center node. Thus, averaging representations of neighbourhood nodes may mix the representation of center node with representations of nodes from other classes. Especially, this issue is serious on networks with heterophily. The performance degradation of existing GNNs on networks with heterophily demonstrates the loss of discriminative information in propagation.

To alleviate this issue, an intuitive and ideal way is to make the propagation in each ego-network just between the nodes from the same class as shown in Figure 1(c). This can be formulated as

where X i ∈ R ( d i +1) × F and H i ∈ R ( d i +1) × F denote the collections of node attributes and representations of ego-network around node v i , i.e. G i . P i ∈ R ( d i +1) × ( d i +1) is the propagation matrix. If the propagation is only between the nodes from the same classes, P i should be block diagonal matrix, whose rank is the same as the number of class in this ego-network. However, it is difficult to known the propagation matrix P i , since the classes of nodes are unknown in self-supervised learning task. Fortunately, it can be obtain that the rank of H i is no more than the number of class in this ego-network, since the rank of P i is the same as the number of class in this ego-network. Therefore, the matrix H i should be low-rank.

3.2 Low-Rank Matrix Decomposition-based GNN

Based on the analysis in pervious subsection, this section proposes the Low-Rank Matrix Decomposition-based GNN (LRD-GNN-Matrix). Motivated by the Robust PCA (RPCA) [21], LRD-GNN-Matrix decomposes the attribute matrix X i into low-rank H i and sparse S i , i.e.,

where rank ( H ) is the rank of the matrix H , || S || 0 is the number of non-zero elements in S , i.e., glyph[lscript] 0 norm, and λ is a parameter for tradeoff between the two terms. Since the rank function and glyph[lscript] 0 norm are nonconvex, we can alternatively minimize its convex surrogate as

where || H || ∗ is the trace norm of H , i.e., the sum of the singular values of H . || S || 1 is the glyph[lscript] 1 norm, i.e. the sum of the absolute value of all elements in S . By applying the Augmented Lagrangian Methods (ALM) [22], the constrained optimization problem in Eqs. (5) can be converted to

where Y i is the Lagrange multipliers for the constraint X i = H i + S i . < Y , X > = tr ( YX ′ ) stands for the inner-product of matrix Y and X . This objective function can be optimized via Alternating Direction Methods of Multipliers (ADMM) [23]. ADMM alternatively updates H i , S i and Y i by fixing others. The algorithm details are shown in Appendix. After the low-rank matrix H ∗ i is obtained, the row in H ∗ i corresponding to the node v i is extracted as h ∗ i as the final representation of node v i .

The proposed LRD-GNN-Matrix possesses two attractive characteristics: (1) Parameter-free: The above low-rank matrix decomposition is parameter-free. Thus, this procedure does not need the supervision label, and can be performed under self-supervised learning. (2) Parallelization: The proposed Low-rank GNNs only need the information of ego-network to obtain representation, and thus are easy to be parallelized. Thus, it is potential to be applied on large graphs.

3.3 Low-Rank Tensor Decomposition-based GNN

The proposed LRD-GNN-Matrix can partly alleviate the issue of discriminative information loss in propagation-based GNNs by low-rank representation pursuit. Unfortunately, it is also difficult to obtain robust and accurate node representations only using information in the ego-network, since the small number of nodes, which belong to the same class as center node, in the ego-network. Taking Figure 1(c) as an example, there are only 3 nodes in the ego-network, which belong to the same class as center node. This issue may be serious on networks with heterophily.

Many recent studies demonstrate that nodes with long distances from the center node can also provide additional information for center node representation. These distant nodes possess very different attributes from the center node, even they belong to the same class. Therefore, propagations with these distant nodes may benefit the robust representation of center node. To this end, this subsection enhances the LRD-GNN-Matrix by proposing Low-Rank Tensor Decomposition-based GNN (LRDGNN-Tensor) as shown in 1(d). LRD-GNN-Tensor constructs the node attribute tensor by selecting similar ego-networks and splicing the attribute matrices of similar ego-networks into 3-way tensor. Analogous to the matrix case in Section 3.2, X i ∈ R ( d i +1) × F × ( M +1) and H i ∈ R ( d i +1) × F × ( M +1) denote the constructed node attribute tensor and representations tensor for ego-network around node

v i , where M is the number of selected similar ego-networks. To obtain robust node representation with the help of selected similar ego-networks, low-rank tensor decomposition is performed on X i

The glyph[lscript] 1 norm || · || 1 , which is the sum of the absolute value of all elements, can be directly extend from matrix to tensor.

Unfortunately, it is difficult to directly extend nuclear norm from matrix to tensor. Motivated by the Tensor RPCA (TRPCA) [24, 25], the tenor nuclear norm of a tensor A ∈ R n 1 × n 2 × n 3 , denoted as || A || ∗ , is defined as the average of the nuclear norm of all the frontal slices of discrete Fourier transformation of A along the 3-rd dimension ¯ A = fft ( A , [] , 3) , i.e. || A || ∗ = 1 n 3 ∑ n 3 i =1 || ¯ A ( i ) || ∗ , where ¯ A ( i ) denotes the frontal slice ¯ A (: , : , i ) . There exist an important property that the block circulant matrix can be mapped to a block diagonal matrix in the Fourier domain, i.e.,

where F n 3 denotes discrete Fourier Transform matrix ⊗ stands for Kronecker product, and

According to this key property, the tensor nuclear norm is essentially equivalent to the matrix nuclear norm of the block circulant matrix, i.e.,

If the selected M similar ego-networks are the ego-networks around nodes v i 1 , ..., v i M , X i is constructed by splicing X i and X i 1 , ..., X i M , and H i is constructed by splicing H i and H i 1 , ..., H i M , then the tensor nuclear norm of H i is

which can capture the long-distance relationship between original ego-networ G i and selected similar ego-networks G i 1 , ..., G i M . Therefore, compared to LRD-GNN-Matrix, LRD-GNN-Tensor can exploit long-distance information, and thus obtain robust and accurate node representation. The detailed algorithm to optimize Eq. (7) is given in Appendix.

4 Experiments

Datasets. Our experiments are conducted on 12 commonly used benchmark datasets, including 6 homophilic graph datasets (i.e., Cora, CiteSeer, PubMed, Wiki-CS, Amazon Computers and Amazon Photo [26, 27, 28]) and 6 heterophilic graph datasets (i.e., Chameleon, Squirrel, Actor, Cornell, Texas, and Wisconsin [29]). The statistics of datasets are summarized in Table 1.

Cora, CiteSeer and PubMed [26] are three citation network datasets, where nodes indicate a paper and each edge indicates a citation relationship between two papers. The labels are the research topic of papers. Wiki-CS [27] is a reference network constructed based on Wikipedia. The nodes correspond to articles about computer science and edges are hyperlinks between the articles. Nodes are labeled with ten classes each representing a branch of the field. Amazon Computers and Amazon Photo [28] are two co-purchase networks from Amazon. In these networks, each node indicates a good, and each edge indicates that two goods are frequently bought together. The labels are the category of goods. Cornell, Texas and Wisconsin [29] are three web page networks from computer science departments of diverse universities, where nodes are web pages and edges are hyperlinks

between two web pages. The labels are types of web pages. Chameleon and Squirrel [29] are two Wikipedia networks where nodes denote web pages in Wikipedia and edges denote links between two pages. The labels stand for the average traffic of the web page.

For Cora, CiteSeer, and PubMed datasets, we adopt the public splits with 20 labeled nodes per class for training, 500 nodes for validation and 1000 nodes for testing. For Wiki-CS, Computers and Photo datasets, we randomly split all nodes into three parts, i.e., 10 % nodes for training, 10 % nodes for validation and the remaining 80 % nodes for testing. The performance on heterophilic datasets is evaluated on the commonly used 48 % /32 % /20 % training/validation/testing.

Baselines. To verify the superiority of the proposed LRD-GNN , We compare it with four groups of baseline methods: (1) The multiple layer perception (MLP) and classic GNN models for node classification task including vanilla GCN [4] and GAT [5]; (2) GNN models designed for alleviating over-smoothing issue or networks with heterophily including JKNet [6], GPR-GNN [10], FAGCN [9] and H2GCN [8]; (3) Conventional self-supervised graph representation learning methods including DeepWalk [30], node2vec [31], GAE and VGAE [11]; (4) Contrastive self-supervised baselines including DGI [13], GMI [14], MVGRL [15], GRACE [16], GCA [17] and BGRL [18].

Experimental details. All methods were implemented in Pytorch with Adam Optimizer. We run 10 times of experiments and report the averaged test accuracy with standard deviation. All the parameters of baselines are tuned to get preferable performance in most situations or the same as authors' original implementations. The hyper-parameter search space is: learning rate in {0.1, 0.05, 0.01}, dropout in {0.2, 0.3, 0.4}. Besides, early stopping with a patience of 200 epochs and L2 regularization with coefficient in {1E-2, 5E-3, 1E-3} are employed to prevent overfitting.

4.1 Experimental Results

4.1.1 Evaluation on node classification task

Performance Comparison. The mean classification accuracy with the standard deviation on 6 homophilic datasets and 6 heterophilic datasets are presented in Table 2 and Table 3, respectively. We























































































compare the proposed LRD-GNN-Matrix and its tensor version LRD-GNN-Tensor with baselines. First of all, we observe that LRD-GNN-Tensor outperforms all baseline methods in 10 out of 12 benchmarks and achieves competitive results on the rest 2 benchmarks.

The superior performance indicates the tensor low-rank decomposition can universally benefit representation learning, since low-rank structure widely exists in diverse real-world graph data. And the construction of tensors effectively captures the connections among similar nodes. Note that LRD-GNN outperforms state-of-the-art deep model JKNet on both homophilic and heterophilic datasets. This demonstrates that shallow-layer information is actually quite abundant for extracting node representation as for LRD-GNN . Additionally, compared with GPR-GNN, FAGNN and H2GCN which are all the GNNs designed for processing datasets with heterophily, we observe that LRD-GNN achieves new state-of-the-art results on all heterophilic datasets. This verifies that the low-rank structure remains the heterophilic information from the ego-networks instead of the impairing heterophilic information as processing by averaging operation.

In Table 3, we find that LRD-GNN significantly outperforms conventional and contrastive selfsupervised methods. The main reason is that these self-supervised methods constantly smooth the representations along heterophilic edges, which destroys the low-rank structure and makes the representations indistinguishable. In contrast, our method overcomes these issues via low-rank tensor decomposition, and learns more expressive representations. Besides, it can be observed that LRDGNN-Tensor outperforms LRD-GNN-Matrix in most cases, which indicates that combining global information from distant nodes with the local is necessary. These results suggest that by adopting the low-rank decomposition in GNNs, our proposed LRD-GNN is more effective and universal than the previous models on processing datasets with both homophily and heterophily for node classification.

The following experiments are all based on LRD-GNN-Tensor, which is abbreviated as LRD-GNN.

























































































Visualization. To provide an intuitive interpretation, we apply t-SNE to visualize the node embeddings obtained by GCN, GAT and LRD-GNN on four datasets. As shown in Figure 4, the clusters of embeddings of nodes from different classes are marked with various colors. By visually analyzing the clusters and distribution of the node classification results, we can find the characteristics of the corresponding models. The clusters of embeddings of different classes processed by GCN are overlapped, which demonstrates that GCN tends to be under-fitting. The clusters of embeddings obtained by GAT are also irregular and sharp, especially the poor performance on Chameleon which indicates that simply aggregating information does not work on homophilic graphs. While the clusters of embedding obtained by LRD-GNN are more regular and the nodes with the same label exhibit spatial clustering, which shows the power of LRD-GNN.

4.1.2 Robustness Analysis

In this experiment, we investigate the robustness of LRD-GNN on graph data. We perturb graph structure and input node attributes by randomly adding noisy edges and attributes respectively, and test the node classification accuracy on the representations learned from perturbed graphs. We compare the performance of LRD-GNN with GCN, GAT and GRACE on Cora, CiteSeer and Pubmed datasets. The classification results under different perturbation rates are shown in Figure 2 and Figure 3. From Figure 2, we can see that LRD-GNN consistently outperforms the other three methods under different perturbation rates. Specifically, the proposed LRD-GNN obtains node representations without propagation and the graph topology is only used to form the ego-network. However, other methods need to propagate the information based on topology, which indicates that they are more likely to be affected by corrupted topology information. Therefore, LRD-GNN tends to be more robust to topology noises than the existing GNNs. Figure 3 reports that LRD-GNN is also superior to other baselines under attributes perturbation, which can be attributed to the denoising ability of low-rank decomposition and recovery in LRD-GNN . These experimental results demonstrate the strong robustness of LRD-GNN against random attacks on graph topology and node attributes.

4.1.3 Ablation Study

To analyze the effectiveness of different tensor construction methods in LRD-GNN , we conduct experiments on several variants of tensor. We compare the performance of LRD-GNN under different tensor construction methods, and node classification results with several variants are shown in Table 4. We mainly investigate methods of constructing tensors from topology and attribute two perspectives: (1) select nodes with similar attributes; (2) select nodes with similar local structures, and then use

Chameleon

CiteSeer

Photo

Computers

GCN

LRD-GNN















































































































































these nodes' ego-network matrix in different orders (i.e., X , AX , A 2 X ) to construct tensors, where each ego-network matrix forms a slice of the target tensor.

From Table 4, we can find that under the same configuration, node embeddings obtained from tensors constructed by nodes with similar attributes are more expressive than tensors constructed by nodes with similar local structures, both on the homophilic graphs (Cora and CiteSeer) and heterophilic graphs (Chameleon, Cornell and Texas). When higher-order information (i.e., A 2 X ) is utilized, the performance on heterophilic datasets is significantly improved. However, the variant of tensor which contains AX is not competitive enough compared to other variants on heterophilic graphs, which indicates that higher-order information is important for representation learning on heterophilic graphs. In addition, we can observe that tensors that preserve the original node attributes outperform tensors formed only by node representations obtained by propagation, which indicates that original node attributes play a prominent role.

4.1.4 Over-smoothing Problem Analysis

To validate whether LRD-GNN can alleviate the over-smoothing problem, we compare the performance of LRD-GNN with several representative self-supervised and semi-supervised GNNs under different model depth. We regard the process of obtaining node representations via TensorRPCA as a layer of LRD-GNN framework. The node classification results on Cora, CiteSeer and Wiki-CS are shown in Figure 5. It can be seen that GCN achieves competitive performance at two layers. As the number of layers increases, the performance of GCN drops rapidly, which indicates that GCN suffers from over-smoothing seriously. GAT alleviates the over-smoothing issue by introducing irrelevant multi-channel propagations. Unfortunately, it also becomes over-smoothing after few layers. DGI and MVGRL, which adopt GCN layer as encoder, also tend to be over-smoothing as depth increases. Instead, the results of LRD-GNN are stable and higher than other methods on different types of networks. The reasons are two-folds: one is that the proposed LRD-GNN obtains node representations without propagation and the graph topology is only used to form the ego-network, which can prevent node representations from being too similar. Another is that the low-rank decom-

position in LRD-GNN reduces the impact of noise, which further keeps node representations from becoming indistinguishable. Through these two designs, when the model goes deep, the performance of LRD-GNN is significantly better than the above self-supervised and semi-supervised methods, which indicates that LRD-GNN has a good capability to alleviate over-smoothing.

5 Conclusions

This paper breaks through the common choice of employing propagation-based GNNs, which aggregate representations of nodes belonging to different classes and tend to lose discriminative information. Firstly, this paper theoretically shows that the obtained representation matrix should follow the low-rank characteristic. Then, to meet this requirement, this paper proposes the Low-Rank Decomposition-based GNNs (LRD-GNN) by employing Low-Rank Decomposition to the attribute matrix. Furthermore, to incorporate long-distance information, Low-Rank Tensor Decompositionbased GNN (LRD-GNN-Tensor) is proposed by constructing the node attribute tensor from selected similar ego-networks and performing Low-Rank Tensor Decomposition. Extensive experiments demonstrate the superior performance and the robustness to noises of the proposed LRD-GNNs.

Acknowledgment

This work was supported in part by the National Natural Science Foundation of China (No. U22B2036, 62376088, 61972442, 62102413, U1936210, U1936208, 11931915), in part by the National Science Fund for Distinguished Young Scholarship of China (No. 62025602), in part by the Natural Science Foundation of Hebei Province of China under Grant F2020202040, in part by the Fok Ying-Tong Education Foundation China (No. 171105), and in part by the Tencent Foundation and XPLORER PRIZE.

References

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. TNNLS, 32(1):4-24, 2021.

Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI Open, 1:57-81, 2020.

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In ICML, pages 1263-1272, 2017.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In ICML, pages 5449-5458, 2018.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML, pages 1725-1735, 2020.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In NeurIPS, 2020.

Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. pages 3950-3957, 2021.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In ICLR, 2021.

Thomas N. Kipf and Max Welling. Variational graph auto-encoders. CoRR, abs/1611.07308, 2016.

Jiwoong Park, Minsik Lee, Hyung Jin Chang, Kyuewang Lee, and Jin Young Choi. Symmetric graph convolutional autoencoder for unsupervised graph representation learning. In ICCV, pages 6518-6527. IEEE, 2019.

Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In ICLR. OpenReview.net, 2019.

Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In WWW, pages 259-270. ACM / IW3C2, 2020.

Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-view representation learning on graphs. In ICML, volume 119, pages 4116-4126. PMLR, 2020.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. CoRR, abs/2006.04131, 2020.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW. ACM, Apr 2021.

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Velickovic, and Michal Valko. Bootstrapped representation learning on graphs. CoRR, abs/2102.06514, 2021.

William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pages 1024-1034, 2017.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.

Emmanuel J. Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? J. ACM, 58(3):11:1-11:37, 2011.

Stephen P. Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn., 3(1):1-122, 2011.

Stephen P. Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2014.

Canyi Lu, Jiashi Feng, Yudong Chen, Wei Liu, Zhouchen Lin, and Shuicheng Yan. Tensor robust principal component analysis: Exact recovery of corrupted low-rank tensors via convex optimization. In CVPR, pages 5249-5257, 2016.

Canyi Lu, Jiashi Feng, Yudong Chen, Wei Liu, Zhouchen Lin, and Shuicheng Yan. Tensor robust principal component analysis with a new tensor nuclear norm. IEEE Trans. Pattern Anal. Mach. Intell., 42(4):925-938, 2020.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine, 29(3):93-93, 2008.

P Mernyei and C Wiki-CS Cangea. A wikipedia-based benchmark for graph neural networks. arxiv 2020. arXiv preprint arXiv:2007.02901, 2007.

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation, 2019.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. In ICLR, 2020.

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: online learning of social representations. In SIGKDD, pages 701-710, 2014.

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In SIGKDD, pages 855-864, 2016.

A Algorithm Description of ADMM

We use Alternating Direction Methods of Multipliers (ADMM) to solve convex optimization problem in RPCA and TensorRPCA , and the details are shown in Algorithm1 and Algorithm2 which are named Matrix ADMM and Tensor ADMM, respectively.

Algorithm 1: Matrix ADMM

end

Algorithm 2: Tensor ADMM

B Select Similar Ego-networks

In section 3.3, we enhance LRD-GNN-Matrix to tensor version LRD-GNN-Tensor. The ego-networks around nodes which are similar with target node are selected to construct tensor. We evaluate similarity from two perspectives. (1) We select nodes which have similar attributes with target node. For instance, the Cosine Similarity is used to measure the similarity of attributes between nodes. (2) We select nodes which have similar local structures with target node. We use the Shannon entropy value of ego-network to measure local structure similarity. For the ego-network G i = ( V i , E i ) around v i , the Shannon entropy value H ( G i ) is defined as

where P ( v ) is the probability of the random walking visiting v in ego-network. Then, we choose several nodes' ego-networks that are close to the Shannon entropy value of the central node to construct tensor.

Besides, to combine information from different orders, we also utilize the selected nodes' egonetworks after propagation(i.e., AX,A 2 X ) as part of the tensor. The specific methods for constructing tenosr and the performance have been presented in section 4.1.3 Ablation Study.

Peer reviews:
comments: SUMMARY:
The paper argues that when dealing with self-supervised learning tasks, utilizing propagation-based GNNs as encoder inevitably encounters two serious issues, i.e., failing to capture local property due to global parameters and lacking the ability on handling homogenous networks without label information. To this end, the authors introduce a different perspective by replacing the propagation-based GNNs with low-rank decomposition-based GNNs. The authors formulate the low-rank decomposition-based GNNs by employing low-rank decomposition to the attribute matrix. Besides, considering that networks require long-distance information for representation, the authors introduce the tensor-based formulation and construct the node attribute tensor from selected similar ego-networks.

STRENGTHS:
- The paper provides a novel perspective to tackle GNNs encoder for self-supervised learning. 
- The proposed low-rank decomposition-based GNNs effectively solve the serious issues introduced by employing propagation-based GNNs. 
- The consideration of incorporating long-distance information successfully captures the long-distance relationships between original and selected similar ego-networks.
- The experiments show the superiority and generalization of the proposed low-rank tensor decomposition-based GNNs. The ablation study and experimental analysis illustrate the success in a complete view.

WEAKNESSES:
- The reason why the low-rank decomposition-based GNNs preserve local information is not presented clearly. 
- Can the authors clarify the differences between networks beyond homophily and network with heterophily?
- How can the proposed model be enhanced in semi-supervised tasks by using the node labels.

QUESTIONS:
see Weaknesses.

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe encoder for self-supervised graph neural networks is investigated.  The authors identify the weaknesses of capturing local property and handling heterophilic networks in existing propagation-based encoder. They observe that the obtained node representations possess low-rank characteristics and tend to meet it using the low-rank matrix factorization. Besides, to extend to incorporate global information, attribute tensor is constructed and low-rank tensor factorization is applied. Experiments shows its effectiveness and robustness to noises, especially on networks with heterophily.\n\nSTRENGTHS:\n•\tThe investigated encoder issues are significant to self-supervised graph neural network. Especially, the global parameter and the lack of supervision information are critical for encoder choice. \n\n•\tThe employed low-rank decomposition algorithms are novel to GNNs. The decomposition to local attribute matrix facilitates the local structure capture and the requirements to label.\n\n•\tThe performance enhancement on heterophilic networks is remarkable. Experiments on robustness and preventing oversmoothing are convincing. \n\n•\tThe presentation is clear. The figure of the proposed framework is elaborate to show the process. The figure of visualization demonstrates the discriminability of node representations.\n\nWEAKNESSES:\n•\tThe relationship between the proposed method and existing self-supervised GNN is not clear. Existing self-supervised learning relays on encoder for supervised learning and an objective function. However, the proposed method does not need objective function. Therefore, it is important to find their connections. \n\n•\tIt is difficult for readers, who are not familiar with low-rank, to get the key points in motivations and details of the proposed methods. It is better to review the concepts of low-rank and self-supervised in preliminaries section or appendix.\n\n•\tThere are some typos. For example, the is a nuclear norm missing in the formula between lines 162 and 163.\n\n•\tThe performance on preventing over-smoothing issue is compared against semi-supervised GNNs, i.e., GCN and GAT. The self-supervised GNNs should be compared with.\n\n•\tThe proposed LRD-GNN is a non-parameter model, which may benefit the task of self-supervised learning. However, how can it be improved if the labels are available.\n\nQUESTIONS:\nRefer to Weaknesses.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to alleviate the issues in propagation-based self-supervised GNN using the low-rank matrix/tensor decompositions. Firstly, it points out existing self-supervised GNNs can not capture local information and handle networks with heterophily due to the global learnable parameters and the lack of supervision information.  Secondly, it investigates the low-rank property of the local representation matrix. Thirdly, to meet these requirements, it presents low-rank matrix/tensor decomposition, which possesses attractive characteristics. Experiments on real networks verify the superior performance and robustness to noises.\n\nSTRENGTHS:\n- The motivation makes sense. It is interesting and meaningful to reveal the issues in existing self-supervised GNNs. Due the shared global parameter and lack of label information, propagation-based encoders tend to have some drawbacks. It seems the first attempt to seek specific encoder for self-supervised GNNs.\n\n- The technology is solid and novel. The observation that local representation matrix should be low-rank is insightful and interesting. The employed low-rank matrix/tensor decomposition are novel to the field of graph neural network. Especially, the tensor decomposition seems technically solid. Besides, the proposed methods have some attractive properties.\n\n- The evaluations are sufficient. Both the employed datasets and the baselines are representative and adequate. The performance improvements are acceptable. The visualization and ablation study are convincing.\n\nWEAKNESSES:\n- Some descriptions are not clear. For example, it is confusing to put the overflows of both matrix decomposition and tensor one in the same figure. Besides, Figure is too small and compact. See questions bellow.\n\n-\tIt is difficult to understand why the proposed methods is better than propagation-based ones for reviewers who are not familiar with low-rank decomposition. It is not as intuitive as in computer vision. Therefore, some preliminaries should be given.\n\n-\tA section on related work should be given to make the readers to judge the contributions of this paper.\n\n-\tThe presentation and writing should be carefully check.\n\nQUESTIONS:\n- What are the differences between unsupervised learning and self-supervised learning.  Although I like the idea of seeking representation using low-rank decomposition, I think it is an unsupervised learning. It may be different from self-supervised one. Could you explain their differences.\n\n- The procedure of tensor construction is not clear. How the similar ego-networks are selected. Besides, it is not clear why the similar ego-networks are concatenated into a tensor instead of a larger matrix.\n\n- It is not clear why the proposed methods are robust to topology and attribute noises. Although the experiments verify this characteristic, the formal explanation should be given.\n\n- What do the white boxes mean in Fig 1(d)?\n\n===\n\nI have read the rebuttal and would like to keep my rating.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the self-supervised graph learning for node classification tasks.\nThe authors make an observation that traditional propagation based GNNs loose discriminative information\nvia node property averaging. To address this issue, the authors propose a novel LRD-GNN method that\nencourages low rank property of the representation matrix. The extended version, LRD-GNN-Tensor, allows\nnode feature propagation between long distance nodes and improve the method's ability to capture long\ndistance relationships. Empirical results show the proposed method outperforms baseline methods and improves\nthe model robustness towards noisy edges.\n\nSTRENGTHS:\n* The authors make an interesting observation that in order to propagate only among same class nodes, the\nproperty matrix needs to be low rank. However, instead of enforcing a low rank decomposition of the\nnode property matrix, the method adopts the Robust PCA relaxation of the rank function/L0 norm and therefore\ndoes not need heuristic hyper-parameters or label supervision.\n\n* Empirical results show the proposed method has better performance than various baseline methods in both homophilic\nand heterophilic settings. Ablation study also shows the proposed method can alleviate the over-smoothing issue.\n\nWEAKNESSES:\n* LRD-GNN-Tensor considers long distance correlations in the graph by grouping similar ego-networks together. However, it is not clear from the paper how are these similar ego-networks selected.\n\n* It is claimed in the paper that the proposed method is scalable yet only small datasets are used in empirical studies.\n\nQUESTIONS:\n* Justification for selecting exactly M similar ego-networks for every node?\n\n* How are the similar ego-networks selected in practice?\n\n* Computation complexity and overall method scalability w.r.t. graph size?\n\nLIMITATIONS:\nThe limitation of this work is not discussed in the paper. \n\nI don't see any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}