SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Liang Yang $^{1}$ , Runjie Shi $^{1}$ , Qiuliang Zhang $^{1}$ , Bingxin Niu $^{1}$ , Zhen Wang $^{2}$ , Xiaochun Cao $^{3}$ , Chuan Wang $^{4*}$
$^{1}$ School of Artificial Intelligence, Hebei University of Technology, Tianjin, China  
 $^{2}$ School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), School of Cybersecurity, Northwestern Polytechnical University, Xi'an, China  
 $^{3}$ School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China  
 $^{4}$ Institute of Information Engineering, CAS, Beijing, China  
yangliang@vip.qq.com, shirunjie2020@163.com, 3463194784@qq.com niubingxin666@163.com, w-zhen@nwpu.edu.cn, caoxiaochun@mail.sysu.edu.cn, wangchuan@jie.ac.cn
Self-supervised learning is introduced to train graph neural networks (GNNs) by employing propagation-based GNNs designed for semi-supervised learning tasks. Unfortunately, this common choice tends to cause two serious issues. Firstly, global parameters cause the model lack the ability to capture the local property. Secondly, it is difficult to handle networks beyond homophily without label information. This paper tends to break through the common choice of employing propagation-based GNNs, which aggregate representations of nodes belonging to different classes and tend to lose discriminative information. If the propagation in each ego-network is just between the nodes from the same class, the obtained representation matrix should follow the low-rank characteristic. To meet this requirement, this paper proposes the Low-Rank Decomposition-based GNNs (LRD-GNN-Matrix) by employing Low-Rank Decomposition to the attribute matrix. Furthermore, to incorporate long-distance information, Low-Rank Tensor Decomposition-based GNN (LRD-GNN-Tensor) is proposed by constructing the node attribute tensor from selected similar ego-networks and performing Low-Rank Tensor Decomposition. The employed tensor nuclear norm facilitates the capture of the long-distance relationship between original and selected similar ego-networks. Extensive experiments demonstrate the superior performance and the robustness of LRD-GNNs.
Graph Neural Networks (GNNs) have been successfully employed by many fields including computer vision (CV), natural language processing (NLP), information retrieval (IR), etc [1, 2, 3]. Vanilla GNNs, such as Graph Convolutional Network (GCN) [4] and Graph Attention Network (GAT) [5], are designed for semi-supervised node classification task, where part of nodes are labeled. To make models adaptive to networks beyond hompohily, flexible GNNs with more trainable parameters are proposed, such as JKNet [6], GCNII [7], H2GCN [8], FAGCN [9] and GPRGNN [10]. The role of label information is more critical for these flexible GNNs.
Inspired by the superior performance in CV and NLP, self-supervised learning, especially contrastive learning, is introduced to training GNNs. While a large number of methods are proposed, they can
be divided into two categories, i.e., contrastive and predictive models. The predictive models, such as Graph AutoEncoder (GAE) and its variational version (VAGE), MGAE [11], GALA [12] and GROVER, require data-label pairs, where the labels are self-generated from the data and significantly different for different tasks. Without the sensitive generated labels, contrastive models, which only require data-data pairs, attract more attention. The representative graph contrastive learning methods include DGI [13], GMI [14], MVGRL [15], GRACE [16], GCA [17] and BGRL [18].
Most existing graph self-supervised learning models employ vanilla GNNs designed for semi-supervised learning with global parameters as encoders. Especially, GCN with one or two layers, where only the feature mappings are trainable, is the most familiar choice. Unfortunately, this common choice tends to cause two serious issues. Firstly, global parameters cause the model lack the ability to capture the local property. For example, the mapping functions in GCN are shared by all nodes. Without the supervision from labels, the reliability of global parameters is weaker than that of the local ones. Secondly, it is difficult to handle networks beyond homophily. Most universal GNNs for networks beyond homophily introduce additional trainable parameters to distinguish neighbourhoods from different classes with the help of labels, such as GPRGNN and FAGNN. Without the label information, this process becomes complicated and fallible.
This paper tends to break through the common choice of employing propagation-based GNNs in self-supervised graph learning. Propagation-based GNNs seek node representation by averaging the representations of nodes, which may belong to different classes, from neighbourhoods, and thus tends to lose discriminative information. To overcome this issue, an intuitive and ideal way is to make the propagation in each ego-network just between the nodes from the same class. By this way, the obtained representation matrix for ego-network should follow the low-rank characteristic. To meet this requirement, this paper proposes the Low-Rank Decomposition-based GNNs (LRD-GNN) by employing Low-Rank Decomposition to the attribute matrix. By directly applying Low-Rank Matrix Decomposition to the attribute matrix of the ego-network, Low-Rank Matrix Decomposition-based GNN (LRD-GNN-Matrix) is given.
Furthermore, to incorporate long-distance information, Low-Rank Tensor Decomposition-based GNN (LRD-GNN-Tensor) is proposed by extending LRD-GNN-Matrix to tensor version. LRD-GNN-Tensor constructs the node attribute tensor by selecting similar ego-networks and splicing the attribute matrices of similar ego-networks into 3-way tensor. Motivated by the Tensor RPCA, tensor nuclear norm is defined as the average of the nuclear norm of all the frontal slices of discrete Fourier transformation of tensor along the 3-rd dimension. This tensor nuclear norm is equivalent to the matrix nuclear norm of the block circulant matrix of the tensor, which facilitates the capture of the long-distance relationship between original ego-network and selected similar ego-networks.
- We point out the necessity of designing specific encoder for self-supervised graph learning.
- We theoretically analyze the low-rank property of the representation matrix in ego-network.
- We proposed Low-Rank Decomposition-based GNNs (LRD-GNN) and its two instances, LRD-GNN-Matrix and LRD-GNN-Tensor.
- We experimentally evaluate the superior performance and the robustness of LRD-GNN.
Notations: In this paper, boldface Euler script letters, e.g.,  $\mathcal{A}$ , are utilized to denote tensors. Matrices are denoted by boldface capital letters, e.g.,  $\mathbf{A}$ ; vectors are denoted by boldface lowercase letters, e.g.,  $\mathbf{a}$ , and scalars are denoted by lowercase letters, e.g.,  $a$ . For a 3-way tensor  $\mathcal{A}$ , its  $(i,j,k)$ -th entry is represented as  $\mathcal{A}_{ijk}$  or  $a_{ijk}$  and use  $\mathcal{A}(i,:,:)$  and  $\mathcal{A}(:,i,:)$  and  $\mathcal{A}(:,i)$  to denote respectively the  $i$ -th horizontal, lateral and frontal slice. The frontal slice  $\mathcal{A}(:,i)$  is denoted compactly as  $\mathbf{A}^{(i)}$ .
Let  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  denote a graph with node set  $\mathcal{V} = \{v_{1},v_{2},\dots ,v_{N}\}$  and edge set  $\mathcal{E}$ , where  $N$  is the number of nodes. The topology of graph  $\mathcal{G}$  can be represented by its adjacency matrix  $\mathbf{A} = [a_{ij}]\in \{0,1\}^{N\times N}$ , where  $a_{ij} = 1$  if and only if there exists an edge  $e_{ij} = (v_i,v_j)$  between nodes  $v_{i}$  and  $v_{j}$ . The degree matrix  $\mathbf{D}$  is a diagonal matrix with diagonal element  $d_{i} = \sum_{i = 1}^{N}a_{ij}$  as the degree of node  $v_{i}$ .  $\mathcal{N}(v_i) = \{v_j|(v_i,v_j)\in \mathcal{E}\}$  stands for the neighbourhoods of node  $v_{i}$ . Let  $\mathcal{G}_i = (\mathcal{V}_i,\mathcal{E}_i)$  represents the ego-network around node  $v_{i}$ , where  $\mathcal{V}_i = \mathcal{N}(v_i)\cup v_i$  and  $\mathcal{E}_i$  denotes
(a) Ego-network
(b) Existing Propagation-based GNNs
(c) Low-Rank Matrix Decomposition-based GNN
Similar Ego-networks Selection
Feature Matrices Extraction
(d) Low-Rank Tensor Decomposition-based GNN
Figure 1: Framework of the proposed Low-Rank Decomposition-based GNNs (LRD-GNNs) and comparison with existing GNNs. (a) One ego-network example. Colors of nodes stand for categories, while center node, i.e. node 1, is highlighted with purple dashed border. (b) Existing propagation-based GNNs are equivalent to weighted averaging of neighbourhood nodes. (c) Low-Rank Matrix Decomposition-based GNN (LRD-GNN-Matrix). The propagations between nodes in the same class tend to make the embedding matrix of node is the ego-network be low-rank. To this end, low-rank matrix decomposition is applied to the feature matrix of nodes in the ego-network. (d) Low-Rank Tensor Decomposition-based GNN (LRD-GNN-Tensor). LRD-GNN-Tensor firstly selects ego-networks, which are similar to the ego-network example. Then, the low-rank tensor decomposition is applied to the tensor, which consists of the feature matrices from above selected ego-networks.
edges between nodes in  $\mathcal{V}_i$ .  $\mathbf{X} \in \mathbb{R}^{N \times F}$  and  $\mathbf{H} \in \mathbb{R}^{N \times F'}$  denote the collections of node attributes and representations with the  $i^{th}$  rows, i.e.,  $\mathbf{x}_i \in \mathbb{R}^F$  and  $\mathbf{h}_i \in \mathbb{R}^{F'}$ , corresponding to node  $v_i$ , where  $F$  and  $F'$  stand for the dimensions of attribute and representation. For convenience,  $\mathbf{X}_i \in \mathbb{R}^{(d_i + 1) \times F}$  and  $\mathbf{H}_i \in \mathbb{R}^{(d_i + 1) \times F'}$  denote collections of attributes and representations of ego-network around  $v_i$ .
Graph Neural Networks: Most of the Graph Neural Networks (GNNs) follow an aggregation-combination strategy [3], where each node representation is iteratively updated by aggregating node representations in the local neighbourhoods and combining the aggregated representations with the node representation itself as
$$
\bar {\mathbf {h}} _ {v} ^ {k} = \text {A G G R E G A T E} ^ {k} \left(\left\{\mathbf {h} _ {u} ^ {k - 1} | u \in \mathcal {N} (v) \right\}\right), \quad \mathbf {h} _ {v} ^ {k} = \text {C O M B I N A T E} ^ {k} \left(\mathbf {h} _ {v} ^ {k - 1}, \bar {\mathbf {h}} _ {v} ^ {k}\right), \tag {1}
$$
where  $\bar{\mathbf{h}}_v^k$  stands for the aggregated representation from local neighbourhoods. Besides of the concatenation based implementation, such as GraphSAGE [19] and H2GCN [8], averaging (or summation) has been widely adopted to implement COMBINATE $^k(\cdot, \cdot)$ , such as GCN [4], GAT [5], GIN [20], etc. Except for the MAX and LSTM implementations in GraphSAGE [19], most of the GNNs utilize averaging function to implement AGGREGATE $^k$ . Therefore, they can be unified as
$$
\mathbf {h} _ {v} ^ {k} = \sigma \left(\left(c _ {v v} ^ {k} \mathbf {h} _ {v} ^ {k - 1} + \sum_ {u \in \mathcal {N} (v)} c _ {u v} ^ {k} \mathbf {h} _ {u} ^ {k - 1}\right) \mathbf {W} ^ {k}\right), \tag {2}
$$
where  $\mathbf{W}^k$  represents the learnable parameters and  $\sigma (\cdot)$  denotes the nonlinear mapping function.
3 Low-Rank Decomposition-based GNN
3.1 Motivations
Existing propagation-based GNNs tend to lose discriminative information. As shown in Figure 1(b) and Eq. (2), propagation-based GNNs seek node representation by averaging the representations of nodes from neighbourhoods. Unfortunately, neighbourhoods may belong to different classes from the center node. Thus, averaging representations of neighbourhood nodes may mix the representation of center node with representations of nodes from other classes. Especially, this issue is serious on networks with heterophily. The performance degradation of existing GNNs on networks with heterophily demonstrates the loss of discriminative information in propagation.
To alleviate this issue, an intuitive and ideal way is to make the propagation in each ego-network just between the nodes from the same class as shown in Figure 1(c). This can be formulated as
$$
\mathbf {H} _ {i} = \mathbf {X} _ {i} \mathbf {P} _ {i}, \tag {3}
$$
where  $\mathbf{X}_i\in \mathbb{R}^{(d_i + 1)\times F}$  and  $\mathbf{H}_i\in \mathbb{R}^{(d_i + 1)\times F}$  denote the collections of node attributes and representations of ego-network around node  $v_{i}$ , i.e.  $\mathcal{G}_i$ .  $\mathbf{P}_i\in \mathbb{R}^{(d_i + 1)\times (d_i + 1)}$  is the propagation matrix. If the propagation is only between the nodes from the same classes,  $\mathbf{P}_i$  should be block diagonal matrix, whose rank is the same as the number of class in this ego-network. However, it is difficult to know the propagation matrix  $\mathbf{P}_i$ , since the classes of nodes are unknown in self-supervised learning task. Fortunately, it can be obtain that the rank of  $\mathbf{H}_i$  is no more than the number of class in this ego-network, since the rank of  $\mathbf{P}_i$  is the same as the number of class in this ego-network. Therefore, the matrix  $\mathbf{H}_i$  should be low-rank.
3.2 Low-Rank Matrix Decomposition-based GNN
Based on the analysis in pervious subsection, this section proposes the Low-Rank Matrix Decomposition-based GNN (LRD-GNN-Matrix). Motivated by the Robust PCA (RPCA) [21], LRD-GNN-Matrix decomposes the attribute matrix  $\mathbf{X}_i$  into low-rank  $\mathbf{H}_i$  and sparse  $\mathbf{S}_i$ , i.e.,
$$
\underset {\mathbf {H} _ {i}, \mathbf {S} _ {i}} {\operatorname {a r g m i n}} \operatorname {r a n k} \left(\mathbf {H} _ {i}\right) + \lambda \left\| \mathbf {S} _ {i} \right\| _ {0} \quad s. t. \quad \mathbf {X} _ {i} = \mathbf {H} _ {i} + \mathbf {S} _ {i} \tag {4}
$$
where  $\text{rank}(\mathbf{H})$  is the rank of the matrix  $\mathbf{H}$ ,  $\| \mathbf{S} \|_0$  is the number of non-zero elements in  $\mathbf{S}$ , i.e.,  $\ell_0$  norm, and  $\lambda$  is a parameter for tradeoff between the two terms. Since the rank function and  $\ell_0$  norm are nonconvex, we can alternatively minimize its convex surrogate as
$$
\underset {\mathbf {H} _ {i}, \mathbf {S} _ {i}} {\operatorname {a r g m i n}} \left\| \mathbf {H} _ {i} \right\| _ {*} + \lambda \left\| \mathbf {S} _ {i} \right\| _ {1} \quad s. t. \quad \mathbf {X} _ {i} = \mathbf {H} _ {i} + \mathbf {S} _ {i} \tag {5}
$$
where  $||\mathbf{H}||_{*}$  is the trace norm of  $\mathbf{H}$ , i.e., the sum of the singular values of  $\mathbf{H}$ .  $||\mathbf{S}||_{1}$  is the  $\ell_{1}$  norm, i.e. the sum of the absolute value of all elements in  $\mathbf{S}$ . By applying the Augmented Lagrangian Methods (ALM) [22], the constrained optimization problem in Eqs. (5) can be converted to
$$
\mathcal {F} \left(\mathbf {H} _ {i}, \mathbf {S} _ {i}, \mathbf {Y} _ {i}, \mu\right) = \left| \left| \mathbf {H} _ {i} \right| \right| _ {*} + \lambda \left| \left| \mathbf {S} _ {i} \right| \right| _ {1} + <   \mathbf {Y} _ {i}, \mathbf {X} _ {i} - \mathbf {H} _ {i} - \mathbf {S} _ {i} > + \frac {\mu}{2} \left| \left| \mathbf {X} _ {i} - \mathbf {H} _ {i} - \mathbf {S} _ {i} \right| \right| _ {F} ^ {2}, \tag {6}
$$
where  $\mathbf{Y}_i$  is the Lagrange multipliers for the constraint  $\mathbf{X}_i = \mathbf{H}_i + \mathbf{S}_i$ .  $<  \mathbf{Y}$ $\mathbf{X} > = tr(\mathbf{Y}\mathbf{X}^{\prime})$  stands for the inner-product of matrix  $\mathbf{Y}$  and  $\mathbf{X}$ . This objective function can be optimized via Alternating Direction Methods of Multipliers (ADMM) [23]. ADMM alternatively updates  $\mathbf{H}_i$ ,  $\mathbf{S}_i$  and  $\mathbf{Y}_i$  by fixing others. The algorithm details are shown in Appendix. After the low-rank matrix  $\mathbf{H}_i^*$  is obtained, the row in  $\mathbf{H}_i^*$  corresponding to the node  $v_{i}$  is extracted as  $\mathbf{h}_i^*$  as the final representation of node  $v_{i}$ .
The proposed LRD-GNN-Matrix possesses two attractive characteristics: (1) Parameter-free: The above low-rank matrix decomposition is parameter-free. Thus, this procedure does not need the supervision label, and can be performed under self-supervised learning. (2) Parallelization: The proposed Low-rank GNNs only need the information of ego-network to obtain representation, and thus are easy to be parallelized. Thus, it is potential to be applied on large graphs.
3.3 Low-Rank Tensor Decomposition-based GNN
The proposed LRD-GNN-Matrix can partly alleviate the issue of discriminative information loss in propagation-based GNNs by low-rank representation pursuit. Unfortunately, it is also difficult to obtain robust and accurate node representations only using information in the ego-network, since the small number of nodes, which belong to the same class as center node, in the ego-network. Taking Figure 1(c) as an example, there are only 3 nodes in the ego-network, which belong to the same class as center node. This issue may be serious on networks with heterophily.
Many recent studies demonstrate that nodes with long distances from the center node can also provide additional information for center node representation. These distant nodes possess very different attributes from the center node, even they belong to the same class. Therefore, propagations with these distant nodes may benefit the robust representation of center node. To this end, this subsection enhances the LRD-GNN-Matrix by proposing Low-Rank Tensor Decomposition-based GNN (LRD-GNN-Tensor) as shown in 1(d). LRD-GNN-Tensor constructs the node attribute tensor by selecting similar ego-networks and splicing the attribute matrices of similar ego-networks into 3-way tensor. Analogous to the matrix case in Section 3.2,  $\mathcal{X}_i\in \mathbb{R}^{(d_i + 1)\times F\times (M + 1)}$  and  $\mathcal{H}_i\in \mathbb{R}^{(d_i + 1)\times F\times (M + 1)}$  denote the constructed node attribute tensor and representations tensor for ego-network around node
$v_{i}$ , where  $M$  is the number of selected similar ego-networks. To obtain robust node representation with the help of selected similar ego-networks, low-rank tensor decomposition is performed on  $\mathcal{X}_i$
$$
\underset {\mathcal {H} _ {i}, \mathcal {S} _ {i}} {\operatorname {a r g m i n}} | | \mathcal {H} _ {i} | | _ {*} + \lambda | | \mathcal {S} _ {i} | | _ {1} \quad s. t. \quad \mathcal {X} _ {i} = \mathcal {H} _ {i} + \mathcal {S} _ {i} \tag {7}
$$
The  $\ell_1$  norm  $\| \cdot \| _1$ , which is the sum of the absolute value of all elements, can be directly extended from matrix to tensor.
Unfortunately, it is difficult to directly extend nuclear norm from matrix to tensor. Motivated by the Tensor RPCA (TRPCA) [24, 25], the tenor nuclear norm of a tensor  $\mathcal{A} \in \mathbb{R}^{n_1 \times n_2 \times n_3}$ , denoted as  $||\mathcal{A}||_*$ , is defined as the average of the nuclear norm of all the frontal slices of discrete Fourier transformation of  $\mathcal{A}$  along the 3-rd dimension  $\bar{\mathcal{A}} = \mathrm{fft}(\mathcal{A}, [], 3)$ , i.e.  $||\mathcal{A}||_* = \frac{1}{n_3} \sum_{i=1}^{n_3} ||\bar{\mathbf{A}}^{(i)}||_*$ , where  $\bar{\mathbf{A}}^{(i)}$  denotes the frontal slice  $\bar{\mathcal{A}}(:, :, i)$ . There exist an important property that the block circulant matrix can be mapped to a block diagonal matrix in the Fourier domain, i.e.,
$$
\left(\mathbf {F} _ {n _ {3}} \otimes \mathbf {I} _ {n _ {1}}\right) \cdot \operatorname {b c i r c} (\boldsymbol {\mathcal {A}}) \cdot \left(\mathbf {F} _ {n _ {3}} ^ {- 1} \otimes \mathbf {I} _ {n _ {2}}\right) = \bar {\mathbf {A}} \tag {8}
$$
where  $\mathbf{F}_{n_3}$  denotes discrete Fourier Transform matrix  $\otimes$  stands for Kronecker product, and
$$
\bar {\boldsymbol {A}} = \mathrm {b d i a g} (\bar {\boldsymbol {A}}) = \left[ \begin{array}{c c c c} \bar {\boldsymbol {A}} ^ {(1)} & & & \\ & \bar {\boldsymbol {A}} ^ {(2)} & & \\ & & \ddots & \\ & & & \bar {\boldsymbol {A}} ^ {(n _ {3})} \end{array} \right], \quad \mathrm {b c i r c} (\boldsymbol {A}) = \left[ \begin{array}{c c c c} \mathbf {A} ^ {(1)} & \mathbf {A} ^ {(n _ {3})} & \dots & \mathbf {A} ^ {(2)} \\ \mathbf {A} ^ {(2)} & \mathbf {A} ^ {(1)} & \dots & \mathbf {A} ^ {(3)} \\ \vdots & \vdots & \ddots & \vdots \\ \mathbf {A} ^ {(n _ {3})} & \mathbf {A} ^ {(n _ {3} - 1)} & \dots & \mathbf {A} ^ {(1)} \end{array} \right].
$$
According to this key property, the tensor nuclear norm is essentially equivalent to the matrix nuclear norm of the block circulant matrix, i.e.,
$$
| | \boldsymbol {\mathcal {A}} | | _ {*} = \frac {1}{n _ {3}} \sum_ {i = 1} ^ {n _ {3}} | | \bar {\boldsymbol {\mathbf {A}}} ^ {(i)} | | _ {*} = \frac {1}{n _ {3}} | | \bar {\boldsymbol {\mathbf {A}}} | | _ {*} = \frac {1}{n _ {3}} | | (\mathbf {F} _ {n _ {3}} \otimes \mathbf {I} _ {n _ {1}}) \cdot \mathrm {b c i r c} (\boldsymbol {\mathcal {A}}) \cdot (\mathbf {F} _ {n _ {3}} ^ {- 1} \otimes \mathbf {I} _ {n _ {2}}) | | _ {*} = \frac {1}{n _ {3}} | | \mathrm {b c i r c} (\boldsymbol {\mathcal {A}}) | | _ {*}.
$$
If the selected  $M$  similar ego-networks are the ego-networks around nodes  $v_{i_1},\dots,v_{i_M}$ ,  $\mathcal{X}_i$  is constructed by splicing  $\mathbf{X}_i$  and  $\mathbf{X}_{i_1},\ldots,\mathbf{X}_{i_M}$ , and  $\mathcal{H}_i$  is constructed by splicing  $\mathbf{H}_i$  and  $\mathbf{H}_{i_1},\ldots,\mathbf{H}_{i_M}$ , then the tensor nuclear norm of  $\mathcal{H}_i$  is
$$
\left.\left|\left| \boldsymbol {\mathcal {H}} _ {i} \right|\right| _ {*} = \frac {1}{M + 1} \right|\left. \right.\left|\left| \operatorname {b c i r c} \left(\boldsymbol {\mathcal {H}} _ {i}\right)\right|\right| _ {*} = \frac {1}{M + 1} \left. \right\rVert\left[\begin{array}{c c c c}\mathbf {H} _ {i}&\mathbf {H} _ {i _ {M}}&\dots&\mathbf {H} _ {i _ {1}}\\\mathbf {H} _ {i _ {1}}&\mathbf {H} _ {i}&\dots&\mathbf {H} _ {i _ {2}}\\\vdots&\vdots&\ddots&\vdots\\\mathbf {H} _ {i _ {M}}&\mathbf {H} _ {i _ {M - 1}}&\dots&\mathbf {H} _ {i},\end{array}\right]\left. \right\rVert_ {*} \tag {9}
$$
which can capture the long-distance relationship between original ego-network  $\mathcal{G}_i$  and selected similar ego-networks  $\mathcal{G}_{i_1},\dots,\mathcal{G}_{i_M}$ . Therefore, compared to LRD-GNN-Matrix, LRD-GNN-Tensor can exploit long-distance information, and thus obtain robust and accurate node representation. The detailed algorithm to optimize Eq. (7) is given in Appendix.
Datasets. Our experiments are conducted on 12 commonly used benchmark datasets, including 6 homophilic graph datasets (i.e., Cora, CiteSeer, PubMed, Wiki-CS, Amazon Computers and Amazon Photo [26, 27, 28]) and 6 heterophilic graph datasets (i.e., Chameleon, Squirrel, Actor, Cornell, Texas, and Wisconsin [29]). The statistics of datasets are summarized in Table 1.
Cora, CiteSeer and PubMed [26] are three citation network datasets, where nodes indicate a paper and each edge indicates a citation relationship between two papers. The labels are the research topic of papers. Wiki-CS [27] is a reference network constructed based on Wikipedia. The nodes correspond to articles about computer science and edges are hyperlinks between the articles. Nodes are labeled with ten classes each representing a branch of the field. Amazon Computers and Amazon Photo [28] are two co-purchase networks from Amazon. In these networks, each node indicates a good, and each edge indicates that two goods are frequently bought together. The labels are the category of goods. Cornell, Texas and Wisconsin [29] are three web page networks from computer science departments of diverse universities, where nodes are web pages and edges are hyperlinks
Table 1: Statistics of datasets
Dataset	Cora	CiteSeer	PubMed	Wiki-CS	Computers	Photo	Chameleon	Squirrel	Actor	Cornell	Texas	Wisconsin
# Nodes	2,708	3,327	19,717	11,701	13,752	7,650	2,277	5,201	7,600	183	183	251
# Edges	5,429	4,732	44,338	216,123	245,861	119,081	36,101	217,073	33,544	295	309	499
# Features	1,433	3,703	500	300	767	745	2,325	2,089	932	1,703	1,703	1,703
# Classes	7	6	3	10	10	8	5	5	5	5	5	5