SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuchen Yan $^{1}$ , Yuzhong Chen $^{2}$ , Huiyuan Chen $^{2}$ , Minghua Xu $^{2}$ , Mahashweta Das $^{2}$ , Hao Yang $^{2}$ , Hanghang Tong $^{1}$
<sup>1</sup>University of Illinois at Urbana Champaign, IL, USA
$^{2}$ Visa Research, CA, USA
{yucheny5,hteong}@illinois.edu
{yuzchen, hchen, mixu, mahdas, haoyang}@visa.com
Finding the proper depth  $d$  of a graph convolutional network (GCN) that provides strong representation ability has drawn significant attention, yet nonetheless largely remains an open problem for the graph learning community. Although noteworthy progress has been made, the depth or the number of layers of a corresponding GCN is realized by a series of graph convolution operations, which naturally makes  $d$  a positive integer ( $d \in \mathbb{N}+$ ). An interesting question is whether breaking the constraint of  $\mathbb{N}+$  by making  $d$  a real number ( $d \in \mathbb{R}$ ) can bring new insights into graph learning mechanisms. In this work, by redefining GCN's depth  $d$  as a trainable parameter continuously adjustable within  $(-\infty, +\infty)$ , we open a new door of controlling its signal processing capability to model graph homophily/heterophily (nodes with similar/dissimilar labels/attributes tend to be inter-connected). A simple and powerful GCN model TEDGCN, is proposed to retain the simplicity of GCN and meanwhile automatically search for the optimal  $d$  without the prior knowledge regarding whether the input graph is homophilic or heterophilic. Negative-valued  $d$  intrinsically enables high-pass frequency filtering functionality via augmented topology for graph heterophily. Extensive experiments demonstrate the superiority of TEDGCN on node classification tasks for a variety of homophilic and heterophilic graphs.
Graph convolutional network (GCN) [28, 50, 19, 47] has exhibited superb power in a myriad of graph learning tasks, such as knowledge graph reasoning [56, 53, 54, 52, 36], network alignment [63, 64, 68, 74, 20, 35, 67], multi-network mining [65, 73, 42, 13, 27, 17], networked time series imputation [51, 25, 55, 16], node recommendation [26, 2, 3], and many more. Since the representation capability of GCN is largely determined by its depth, i.e., the number of graph convolution layers, tremendous research efforts have been made on finding the optimal depth that strengthens the model's ability for downstream tasks. Upon increasing the depth, the over-smoothing issue arises: a GCN's performance is deteriorated if its depth exceeds a certain threshold [28]. It is found in [32] that a graph convolution operation is a special form of Laplacian smoothing [48]. Thus, the similarity between the graph node embeddings grows with the depth so that these embeddings eventually become indistinguishable. Various techniques have been developed to alleviate this issue, e.g., applying pairwise normalization can make distant nodes dissimilar [69], and dropping sampled edges during training slows down the growth of embedding smoothness as the depth increases [43].
Other than the over-smoothing issue due to a large GCN depth, another fundamental phenomenon widely existing in real-world graphs is homophily and heterophily. In a homophilic graph, nodes with
similar labels or attributes tend to be inter-connected, while in a heterophilic graph, connected nodes usually have distinct labels or dissimilar attributes. Most graph neural networks (GNNs) are developed based on the homophilic assumption [66], while models able to perform well on heterophilic graphs often need special treatment and complex designs [4, 76, 62]. Despite the achievements made by these methodologies, little correlation has been found between the adopted GNN model's depth and its capability of characterizing graph heterophily.
For almost any GNN model, the depth needs to be manually set as a hyper-parameter before training, and finding the proper depth usually requires a considerable amount of trials or good prior knowledge of the graph dataset. Since the depth represents the number of graph convolution operations and naturally takes only positive integer values, little attention has been paid to the question of whether a non-integer depth is realizable, and if yes, whether it is practically meaningful, and what kind of unique advantages it can bring to current graph learning mechanisms.
This work revisits the GCN depth from spectral and spatial perspectives and explains the intrinsic connections between the following key ingredients in graph learning: (i) the depth of a GCN, (ii) the spectrum of the graph signal, and (iii) the homophily/heterophily of the underlying graph. Firstly, through eigen-decomposition of the symmetrically normalized graph Laplacian, we present the connection between graph homophily/heterophily and the eigenvector frequencies. Secondly, by introducing the concept of eigengraph, we show the graph topology is equivalent to a weighted linear combination of eigengraphs, and the weights determine the GCN's capability of capturing homophilic/heterophilic graph signals. Thirdly, we reveal that the eigengraph weights can be controlled by GCN's depth, so that an automatically tunable depth parameter is needed to adjust the eigengraph weights into the designated distribution in match of the underlying graph homophily/heterophily.
To realize the adaptive GCN depth, we extend its definition from a positive integer to an arbitrary real number with theoretical feasibility guarantees from functional calculus [45]. With a trainable depth parameter, we propose a simple and powerful model, Trainable Depth-GCN (TEDGCN), with two variants. Extensive experiments demonstrate TEDGCN's ability of automatically searching for the optimal depth, and it is found that negative-valued depth plays the key role in handling heterophilic graphs. Systematical investigation on the optimal depth is conducted via various real-world and synthetic datasets. It in turn inspires the development of a novel graph augmentation methodology. With clear geometric interpretability, the augmented graph structure possesses supreme advantages over the raw input topology, especially for graphs with heterophily. The main contributions of this paper are summarized as follows:
- Analysis. The intrinsic relationship between the negative GCN depth and graph heterophily is discovered; In-depth geometric and spectral explanations are presented.
- Problem. A novel problem of automatic GCN depth tuning for graph homophily/heterophily detection is formulated. To our best knowledge, this work presents the first trial to make GCN's depth negative and trainable by redefining it on the real number domain.
- Model. A simple and powerful model TEDGCN with two variants (TEDGCN-S and TEDGCN-D) is proposed. A novel graph augmentation method is discussed.
- Experiments. Our model achieves superior performance on semi-supervised node classification tasks on 11 graph datasets.
Notations. We utilize bold uppercase letters for matrices (e.g.,  $\mathbf{A}$ ), bold lowercase letters for column vectors (e.g.,  $\mathbf{u}$ ) and lowercase letters for scalars (e.g.,  $\alpha$ ). We use the superscript  $\top$  for the transpose of matrices and vectors (e.g.,  $\mathbf{A}^{\top}$  and  $\mathbf{u}^{\top}$ ). An attributed undirected graph  $\mathcal{G} = \{\mathbf{A}, \mathbf{X}\}$  contains an adjacency matrix  $\mathbf{A} \in \mathbb{R}^{n \times n}$  and an attribute matrix  $\mathbf{X} \in \mathbb{R}^{n \times q}$  with the number of nodes  $n$  and the dimension of node attributes  $q$ .  $\mathbf{D}$  denotes the diagonal degree matrix of  $\mathbf{A}$ . The adjacency matrix with self-loops is given by  $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$  ( $\mathbf{I}$  is the identity matrix), and all variables derived from  $\tilde{\mathbf{A}}$  are decorated with symbol  $\tilde{\cdot}$ , e.g.,  $\tilde{\mathbf{D}}$  represents the diagonal degree matrix of  $\tilde{\mathbf{A}}$ .  $\mathbf{M}^d$  stands for the  $d$ -th power of matrix  $\mathbf{M}$ , while the parameter and node embedding matrices in the  $d$ -th layer of a GCN are denoted by  $\mathbf{W}^{(d)}$  and  $\mathbf{H}^{(d)}$ .
Figure 1: Decompose the symmetrically normalized adjacency matrix into three eigengraphs.
Graph convolutional network (GCN) and simplified graph convolutional network (SGC). The layer-wise message-passing and aggregation of GCN [28] is given by
$$
\mathbf {H} ^ {(d + 1)} = \sigma \left(\tilde {\mathbf {D}} ^ {- \frac {1}{2}} \tilde {\mathbf {A}} \tilde {\mathbf {D}} ^ {- \frac {1}{2}} \mathbf {H} ^ {(d)} \mathbf {W} ^ {(d)}\right), \tag {1}
$$
where  $\mathbf{H}^{(d)} / \mathbf{H}^{(d + 1)}$  stands for the embedding matrix  $(\mathbf{H}^{(0)} = \mathbf{X})$  in the  $d$ -th/  $(d + 1)$ -th layer;  $\mathbf{W}^{(d)}$  is the trainable parameter matrix; and  $\sigma (\cdot)$  is the non-linear activation function. With  $\sigma (\cdot)$  removed in each layer, SGC [59] is obtained as below:
$$
\mathbf {H} ^ {(d)} = \tilde {\mathbf {S}} ^ {d} \mathbf {X} \mathbf {W}, \tag {2}
$$
where  $\tilde{\mathbf{S}} = \tilde{\mathbf{D}}^{-\frac{1}{2}}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-\frac{1}{2}}$ , and the parameter of each layer  $\mathbf{W}^{(i)}$  are compressed into one trainable  $\mathbf{W} = \prod_{i=0}^{d-1}\mathbf{W}^{(i)}$ .
Graph Laplacian and spectrum. In graph theory, graph Laplacian  $\mathbf{L} = \mathbf{D} - \mathbf{A}$  and its symmetrically normalized correspondence  $\mathbf{L}_{sym} = \mathbf{I} - \mathbf{D}^{-\frac{1}{2}}\mathbf{A}\mathbf{D}^{-\frac{1}{2}}$  possess critical properties of the underlying graph  $\mathcal{G}$ .  $\mathbf{L}_{sym}$  has eigenvalues  $[\lambda_1,\lambda_2,\dots ,\lambda_n]$ , where  $\lambda_{i}\in [0,2)$ ,  $\forall i\in \{1,2,\ldots ,n\}$  [11].<sup>1</sup> Here, they are put in the ascending order:  $0 = \lambda_{1}\leq \lambda_{2}\leq \dots \leq \lambda_{n} < 2$ . It can be eigen-decomposed as:  $\mathbf{L}_{sym} = \mathbf{U}\boldsymbol {\Lambda}\mathbf{U}^{\top}$ , where  $\mathbf{U} = [\mathbf{u}_1,\mathbf{u}_2,\dots ,\mathbf{u}_n]$  is the eigenvector matrix  $(\mathbf{u}_i\perp \mathbf{u}_j,\forall i\neq j)$ , and  $\pmb{\Lambda}$  is the diagonal eigenvalue matrix. For each eigenvector  $\mathbf{u}_i$ , we have  $\mathbf{u}_i\mathbf{u}_i^\top \in \mathbb{R}^{n\times n}$ . As we will show in Section 3, this  $n\times n$  matrix can be viewed as the weighted adjacency matrix of a graph with possible negative edges, which we name as the  $i$ -th eigraph of  $\mathcal{G}$ . Accordingly,  $\mathbf{L}_{sym}$  can be written as the linear combination of all eigengraphs weighted by the corresponding eigenvalues [11]:
$$
\mathbf {L} _ {s y m} = \lambda_ {1} \mathbf {u} _ {1} \mathbf {u} _ {1} ^ {\top} + \dots + \lambda_ {i} \mathbf {u} _ {i} \mathbf {u} _ {i} ^ {\top} + \dots + \lambda_ {n} \mathbf {u} _ {n} \mathbf {u} _ {n} ^ {\top}, \tag {3}
$$
where the first eigenvalue  $\lambda_{1} = 0$  [46]. Thus, for SGC, we have
$$
\tilde {\mathbf {S}} = \mathbf {I} - \tilde {\mathbf {L}} _ {s y m} = \tilde {\mathbf {U}} (\mathbf {I} - \tilde {\boldsymbol {\Lambda}}) \tilde {\mathbf {U}} ^ {\top} = \sum_ {i = 0} ^ {n} (1 - \tilde {\lambda} _ {i}) \tilde {\mathbf {u}} _ {i} \tilde {\mathbf {u}} _ {i} ^ {\top}. \tag {4}
$$
A SGC with  $d$  layers requires  $d$  consecutive graph convolution operations, which involves the multiplication of  $\tilde{\mathbf{S}}$  by  $d$  times. Due to the orthogonality of  $\tilde{\mathbf{U}}$ , namely,  $\tilde{\mathbf{U}}^{\top}\tilde{\mathbf{U}} = \mathbf{I}$ , we obtain
$$
\tilde {\mathbf {S}} ^ {d} = \tilde {\mathbf {U}} (\mathbf {I} - \tilde {\boldsymbol {\Lambda}}) \tilde {\mathbf {U}} ^ {\top} \tilde {\mathbf {U}} (\mathbf {I} - \tilde {\boldsymbol {\Lambda}}) \tilde {\mathbf {U}} ^ {\top} \dots \tilde {\mathbf {U}} (\mathbf {I} - \tilde {\boldsymbol {\Lambda}}) \tilde {\mathbf {U}} ^ {\top} = \tilde {\mathbf {U}} (\mathbf {I} - \tilde {\boldsymbol {\Lambda}}) ^ {d} \tilde {\mathbf {U}} ^ {\top} = \sum_ {i = 1} ^ {n} (1 - \tilde {\lambda} _ {i}) ^ {d} \tilde {\mathbf {u}} _ {i} \tilde {\mathbf {u}} _ {i} ^ {\top}, \tag {5}
$$
where  $1 - \tilde{\lambda}_i\in (-1,1]$ , and the depth  $d$  of SGC serves as the power of  $\tilde{\mathbf{S}}$  's eigenvalues.  $\tilde{\mathbf{S}}^d$  can be viewed as the sum of eigengraphs  $\tilde{\mathbf{u}}_i\tilde{\mathbf{u}}_i^\top$  weighted by coefficients  $(1 - \tilde{\lambda}_i)^d$ .
Graph homophily and heterophily. Various homophily/heterophily measures are developed to capture how edges tend to link nodes with the same labels and similar attributes, such as edge homophily[1, 76, 75, 37], node homophily[41], and class homophily[34]. Without the loss of generality, we focus on one mostly used homophily metric, the edge homophily:  $h(\mathcal{G}) = \frac{\sum_{i,j,\mathbf{A}[i,j] = 1}\langle\mathbf{y}[i] = \mathbf{y}[j]\rangle}{\sum_{i,j}\mathbf{A}[i,j]} \in [0,1]$ , where  $\langle x\rangle = 1$  if  $x$  is true and 0 otherwise. A graph is more homophilic for  $h(\mathcal{G})$  closer to 1 or more heterophilic for  $h(\mathcal{G})$  closer to 0.
This work focuses on connected graph without bipartite components (i.e., a connected component which is a bipartite graph).
Overview. Firstly, we establish the intrinsic relationship between graph spectrum and graph heterophily. Secondly, we show how the positive/negative depth  $d$  of a GCN affects the eigengraph weights which in turn impacts the algorithm's capability to capture homophilic/heterophilic graph signals. Thirdly, with the help of functional calculus [45], we present the theoretical feasibility of extending the domain of  $d$  from  $\mathbb{N} +$  to  $\mathbb{R}$ . Finally, by making  $d$  a trainable parameter, we present our model TEDGCN and its variants, which are capable of automatically detecting the homophily/heterophily of the input graph and finding the optimal depth.
Eigenvector frequency and entry deviation. The frequency of graph Laplacian eigenvector  $\mathbf{u}_i$  reflects how much the  $j$ -th entry  $\mathbf{u}_i[j]$  deviates from the  $k$ -th entry  $\mathbf{u}_i[k]$  for each connected node pair  $v_j$  and  $v_k$  in  $\mathcal{G}$  [46]. This deviation is measured by the set of zero crossings of  $\mathbf{u}_i$ :  $\mathcal{Z}(\mathbf{u}_i) := \{e = (v_j, v_k) \in \mathcal{E} : \mathbf{u}_i[j]\mathbf{u}_i[k] < 0, j \neq k\}$ , where  $\mathcal{E}$  is the set of edges in graph  $\mathcal{G}$ . Larger/smaller  $|\mathcal{Z}(\mathbf{u}_i)|$  indicates higher/ lower eigenvector frequency. A zero-crossing also corresponds a negative weighted edge in an eigraph. Due to the widely existing positive correlation between  $\lambda_i$  and  $|\mathcal{Z}(\mathbf{u}_i)|$  [46], large/small eigenvalues mostly correspond to the high/low frequencies of the related eigenvectors. As illustrated by the toy example of  $n = 3$  in Figure 1, for  $\lambda_1 = 0$ , we have  $|\mathcal{Z}(\mathbf{u}_1)| = 0$ , and eigraph  $\mathbf{u}_1\mathbf{u}_1^\top$  is fully-connected; negative edge weights exist in the 2nd and 3rd eigengraphs, indicating more zero crossings ( $|\mathcal{Z}(\mathbf{u}_2)| = 1$  and  $|\mathcal{Z}(\mathbf{u}_3)| = 2$ ) and higher eigenvector frequencies.
Entry deviation and graph heterophily. Reflected by entry deviations, eigenvector frequency and zero-crossings determine the number of negatively weighted edges  $(\mathbf{u}_i[j]\mathbf{u}_i[k] < 0, j \neq k)$  in a corresponding eigengraph  $\mathbf{u}_i\mathbf{u}_i^\top$ . For a GCN/SGC, conducting message-passing along negatively weighted edges makes the embeddings of the connected node pairs  $(v_j, v_k)$  dissimilar. Naturally, higher embedding dissimilarity is more likely to render different label predictions between the node pair, and this generates heterophily. Thus, eigenvectors with higher frequencies and accordingly higher entry deviations contribute more to heterophilic graph signals, and vice versa.
Graph heterophily and GCN's depth. Based on above analyses, high frequency eigenvectors and their corresponding eigengraphs have advantage on capturing graph heterophily. A graph neural network should be able to give high frequency eigengraphs larger weights when modeling heterophilic graphs, whereas low frequency ones should carry larger weights when dealing with homophilic graphs. The weights of eigengraphs are controlled by GCN/SGC's depth  $d$  as follows. For a SGC of depth  $d$ , the weight of the  $i$ -th eigengraph is  $(1 - \tilde{\lambda}_i)^d$ , and changing the depth  $d$  of SGC adjusts the weights of different eigengraphs. Therefore, depth  $d$  controls the model's capability of effectively filtering low/high-frequency signals for graph homophily/heterophily.
Redefined GCN's depth. A question is naturally raised: instead of manually setting the depth  $d$ , can  $d$  be built into the model as a trainable parameter? If yes, a proper set of the eigengraph weights matching the graph homophily/heterophily can be automatically reached by finding the optimal  $d$  in an end-to-end fashion during training.
Differentiable variables need continuity, which requires the extension of depth  $d$  from the discrete positive integer domain  $(\mathbb{N}+)$  to the continuous real number domain  $\mathbb{R}$ . According to functional calculus [45], applying an arbitrary matrix function  $f$  on a graph Laplacian  $\mathbf{L}_{sym}$  is equivalent to applying the same function only on the eigenvalue matrix  $\Lambda$ :
$$
f \left(\mathbf {L} _ {\text {s y m}}\right) = \mathbf {U} f (\boldsymbol {\Lambda}) \mathbf {U} ^ {\top} = \mathbf {U} \left[ \begin{array}{c c c} f \left(\lambda_ {1}\right) & \dots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \dots & f \left(\lambda_ {n}\right) \end{array} \right] \mathbf {U} ^ {\top}, \tag {6}
$$
which also applies to  $\tilde{\mathbf{L}}_{sym}$  and  $\tilde{\mathbf{S}}$ . Armed with this, we seek to realize an arbitrary depth SGC via a power function as  $f(\tilde{\mathbf{S}}) = \tilde{\mathbf{S}}^d = \tilde{\mathbf{U}} (\mathbf{I} - \tilde{\boldsymbol{\Lambda}})^d\tilde{\mathbf{U}}^\top = \sum_{i = 1}^n (1 - \tilde{\lambda}_i)^d\tilde{\mathbf{u}}_i\tilde{\mathbf{u}}_i^\top (d\in \mathbb{R})$ . However, since  $\tilde{\lambda}_i\in [0,2)$ , we have  $(1 - \tilde{\lambda}_i)\leq 0$  when  $1\leq \tilde{\lambda}_i < 2$ , and for  $(1 - \tilde{\lambda}_i)$  taking zero or negative values,  $(1 - \tilde{\lambda}_i)^d$  is not well-defined or involving complex-number-based calculations for a real-valued  $d$  (e.g.,  $(-0.5)^{\frac{3}{8}}$ ) [45]. Moreover, even for integer-valued  $ds$  under which  $(1 - \tilde{\lambda}_i)^d$  is easy to compute, the behavior of  $(1 - \tilde{\lambda}_i)^d$  is complicated versus  $\tilde{\lambda}_i$  and diverges when  $\tilde{\lambda}_i = 1$  for negative  $ds$ , as shown in Figure 2a. Thus, the favored weight distribution may be hard to obtain by tuning  $d$ .
To avoid such complications and alleviate the difficulties for manipulating the eigengraph weights, a transformation function  $g(\cdot)$  operating on the graph Laplacian  $\mathbf{L}_{sym}$  or  $\tilde{\mathbf{L}}_{sym}$  is in need to shift  $g(\lambda_i)$  or  $g(\tilde{\lambda}_i)$  into a proper value range so that its power of a real-valued  $d$  is easy to obtain and well-behaved versus  $\lambda_i$  or  $\tilde{\lambda}_i$ . Without the loss of generality, our following analysis focuses on  $\mathbf{L}_{sym}$  and  $\lambda_i$ . There may exist multiple choices for  $g(\cdot)$  satisfying the requirements. In this work, we focus on the following transformation function:
$$
\hat {\mathbf {S}} = g \left(\mathbf {L} _ {s y m}\right) = \frac {1}{2} \left(2 \mathbf {I} - \mathbf {L} _ {s y m}\right). \tag {7}
$$
This choice of  $g(\cdot)$  holds three properties: (1) Positive eigenvalues. Since we have  $\mathbf{L}_{sym}$ 's  $i$ -th eigenvalue  $\lambda_{i} \in [0,2)$ , the corresponding eigenvalue of  $\hat{\mathbf{S}}$  is  $g(\lambda_{i}) = \frac{1}{2}(2 - \lambda_{i}) \in (0,1]$ . Thus, the  $d$ -th power of  $g(\lambda_{i})$  is computable for any  $d \in \mathbb{R}$ . (2) Monotonicity versus eigenvalues  $\lambda$ . As shown in Figure 2b,  $g(\lambda_{i})^{d} = (1 - \frac{1}{2}\lambda)^{d}$  is monotonically increasing/decreasing when  $\lambda$  varies between 0 and 2 under negative/positive depth. (3) Geometric interpretability. Filter  $\hat{\mathbf{S}}$  can be expressed as:
(a)  $(1 - \tilde{\lambda})^d$
(b)  $(1 - \frac{1}{2}\lambda)^d$
Figure 2: Eigraph weight versus eigenvalue for (a) SGC and (b) TEDGCN-S under different depth  $ds$ .
$$
\hat {\mathbf {S}} = \mathbf {U} \hat {\boldsymbol {\Lambda}} \mathbf {U} ^ {\top} = \mathbf {U} (\mathbf {I} - \frac {1}{2} \boldsymbol {\Lambda}) \mathbf {U} ^ {\top} = \frac {1}{2} \mathbf {I} + \frac {1}{2} (\mathbf {I} - \mathbf {L} _ {s y m}) = \frac {1}{2} (\mathbf {I} + \mathbf {D} ^ {- \frac {1}{2}} \mathbf {A} \mathbf {D} ^ {- \frac {1}{2}}). \tag {8}
$$
As shown in Figure 3, in the spatial domain,  $\hat{\mathbf{S}}$  is obtained via 3 operations on adjacency matrix  $\mathbf{A}$ : normalization, adding self-loops, and scaling all edge weights by  $\frac{1}{2}$  (a type of lazy random walk [39]), while  $\tilde{\mathbf{S}}$  in vanilla GCN/SGC contains 2 operations: adding self-loops and normalization.
With the help of transformation  $g$ , the depth  $d$  is redefined on real number domain, and the message propagation process of depth  $d$  can be realized via the following steps: (1) Eigen-decompose  $\mathbf{L}_{sym}$ ; (2) Calculate  $\hat{\mathbf{S}}^d$  via weight  $g(\lambda_i)^d$  and the weighted sum of all eigengraphs:  $\hat{\mathbf{S}}^d = \mathbf{U}\hat{\mathbf{\Lambda}}^d\mathbf{U}^\top = \sum_{i=1}^{n}g(\lambda_i)^d\mathbf{u}_i\mathbf{u}_i^\top$  (3) Multiply  $\hat{\mathbf{S}}^d$  with original node attributes  $\mathbf{X}$ .
Negative depth explained. First, an intuitive explanation of negative  $d$  can be obtained from the perspective of matrix inverse and message diffusion process when  $d$  takes integer values. Since  $\hat{\mathbf{S}}^{-1}\hat{\mathbf{S}} = \mathbf{U}\hat{\mathbf{A}}^{-1}\mathbf{U}^{\top}\mathbf{U}\hat{\mathbf{A}}^{1}\mathbf{U}^{\top} = \mathbf{I}$ ,  $\hat{\mathbf{S}}^{-1}$  is the inverse matrix of  $\hat{\mathbf{S}}$ . In diffusion dynamics,  $\mathbf{X}$  can be viewed as an intermediate state generated in a series of message propagation steps.  $\hat{\mathbf{S}}\mathbf{X}$  effectively propagates the message one-step forward, while  $\hat{\mathbf{S}}^{-1}$  can cancel the effect of  $\hat{\mathbf{S}}$  on  $\mathbf{X}$  and recover the original message by moving backward:  $\hat{\mathbf{S}}^{-1}\hat{\mathbf{S}}\mathbf{X} = \mathbf{X}$ . Accordingly,  $\hat{\mathbf{S}}^{-1}\mathbf{X}$  traces back to the message's previous state in
the series. Non-integer  $d$  indicates the back- or forward propagation can be a continuous process. Second, as the depth becomes negative, it primarily keeps the eigengraphs of high frequency (See Figure 2b), which have more negative edges. Therefore, it is equivalent to turning some positive edges in the original graph into negative edges in the augmented graph, which in turn makes node embeddings dissimilar in the message-passing process and thus naturally handles the heterophilic graphs (See Figure 1). More discussions on the impact of negative depth in spatial domain are presented in Section 4.4.
TEDGCN-S algorithm. By further making  $d$  a trainable parameter, we present our model, Trainable Depth-GCN-Single (TEDGCN-S), whose final node embedding matrix is given by
$$
\mathbf {H} = \sigma (\hat {\mathbf {S}} ^ {d} \mathbf {X} \mathbf {W}), \tag {9}
$$
Figure 3: The difference between  $\tilde{\mathbf{S}}$  (left) for GCN/SGC and  $\hat{\mathbf{S}}$  (right) for TEDGCN.
forward propagation can be a continuous process. Clearly, it is equivalent to turning some positive in the augmented graph, which in turn makes node process and thus naturally handles the heterophilic the impact of negative depth in spatial domain are
$$
(\hat {\mathbf {S}} ^ {d} \mathbf {X} \mathbf {W}), \tag {9}
$$