Title: From Trainable Negative Depth to Edge Heterophily in Graphs
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph convolutional network
Abstract: Finding the proper depth $d$ of a graph convolutional network (GCN) that provides strong representation ability has drawn significant attention, yet nonetheless  largely  remains an open problem for the graph learning community. Although noteworthy progress has been made, the depth or the number of layers of a corresponding GCN is realized by a series of graph convolution operations, which naturally makes $d$ a positive integer ($d \in \mathbb{N}+$). An interesting question is whether breaking the constraint of $\mathbb{N}+$ by making $d$ a real number ($d \in \mathbb{R}$) can bring new insights into graph learning mechanisms. In this work, by redefining GCN's depth $d$ as a trainable parameter continuously adjustable within $(-\infty,+\infty)$, we open a new door of controlling its signal processing capability to model graph homophily/heterophily (nodes with similar/dissimilar labels/attributes tend to be inter-connected). A simple and powerful GCN model TEDGCN, is proposed to retain the simplicity of GCN and meanwhile automatically search for the optimal $d$ without the prior knowledge regarding whether the input graph is homophilic or heterophilic. Negative-valued $d$ intrinsically enables high-pass frequency filtering functionality via augmented topology for graph heterophily. Extensive experiments demonstrate the superiority of TEDGCN on node classification tasks for a variety of homophilic and heterophilic graphs.

From Trainable Negative Depth to Edge Heterophily in Graphs

Abstract

Finding the proper depth d of a graph convolutional network (GCN) that provides strong representation ability has drawn significant attention, yet nonetheless largely remains an open problem for the graph learning community. Although noteworthy progress has been made, the depth or the number of layers of a corresponding GCN is realized by a series of graph convolution operations, which naturally makes d a positive integer ( d ∈ N + ). An interesting question is whether breaking the constraint of N + by making d a real number ( d ∈ R ) can bring new insights into graph learning mechanisms. In this work, by redefining GCN's depth d as a trainable parameter continuously adjustable within ( -∞ , + ∞ ) , we open a new door of controlling its signal processing capability to model graph homophily/heterophily (nodes with similar/dissimilar labels/attributes tend to be inter-connected). A simple and powerful GCN model TEDGCN, is proposed to retain the simplicity of GCN and meanwhile automatically search for the optimal d without the prior knowledge regarding whether the input graph is homophilic or heterophilic. Negative-valued d intrinsically enables high-pass frequency filtering functionality via augmented topology for graph heterophily. Extensive experiments demonstrate the superiority of TEDGCN on node classification tasks for a variety of homophilic and heterophilic graphs.

1 Introduction

Graph convolutional network (GCN) [28, 50, 19, 47] has exhibited superb power in a myriad of graph learning tasks, such as knowledge graph reasoning [56, 53, 54, 52, 36], network alignment [63, 64, 68, 74, 20, 35, 67], multi-network mining [65, 73, 42, 13, 27, 17], networked time series imputation [51, 25, 55, 16], node recommendation [26, 2, 3], and many more. Since the representation capability of GCN is largely determined by its depth, i.e., the number of graph convolution layers, tremendous research efforts have been made on finding the optimal depth that strengthens the model's ability for downstream tasks. Upon increasing the depth, the over-smoothing issue arises: a GCN's performance is deteriorated if its depth exceeds a certain threshold [28]. It is found in [32] that a graph convolution operation is a special form of Laplacian smoothing [48]. Thus, the similarity between the graph node embeddings grows with the depth so that these embeddings eventually become indistinguishable. Various techniques have been developed to alleviate this issue, e.g., applying pairwise normalization can make distant nodes dissimilar [69], and dropping sampled edges during training slows down the growth of embedding smoothness as the depth increases [43].

Other than the over-smoothing issue due to a large GCN depth, another fundamental phenomenon widely existing in real-world graphs is homophily and heterophily. In a homophilic graph, nodes with

similar labels or attributes tend to be inter-connected, while in a heterophilic graph, connected nodes usually have distinct labels or dissimilar attributes. Most graph neural networks (GNNs) are developed based on the homophilic assumption [66], while models able to perform well on heterophilic graphs often need special treatment and complex designs [4, 76, 62]. Despite the achievements made by these methodologies, little correlation has been found between the adopted GNN model's depth and its capability of characterizing graph heterophily.

For almost any GNN model, the depth needs to be manually set as a hyper-parameter before training, and finding the proper depth usually requires a considerable amount of trials or good prior knowledge of the graph dataset. Since the depth represents the number of graph convolution operations and naturally takes only positive integer values, little attention has been paid to the question of whether a non-integer depth is realizable, and if yes, whether it is practically meaningful, and what kind of unique advantages it can bring to current graph learning mechanisms.

This work revisits the GCN depth from spectral and spatial perspectives and explains the intrinsic connections between the following key ingredients in graph learning: (i) the depth of a GCN, (ii) the spectrum of the graph signal, and (iii) the homophily/heterophily of the underlying graph. Firstly, through eigen-decomposition of the symmetrically normalized graph Laplacian, we present the connection between graph homophily/heterophily and the eigenvector frequencies. Secondly, by introducing the concept of eigengraph, we show the graph topology is equivalent to a weighted linear combination of eigengraphs, and the weights determine the GCN's capability of capturing homophilic/heterophilic graph signals. Thirdly, we reveal that the eigengraph weights can be controlled by GCN's depth, so that an automatically tunable depth parameter is needed to adjust the eigengraph weights into the designated distribution in match of the underlying graph homophily/heterophily.

To realize the adaptive GCN depth, we extend its definition from a positive integer to an arbitrary real number with theoretical feasibility guarantees from functional calculus [45]. With a trainable depth parameter, we propose a simple and powerful model, Trainable Depth-GCN (TEDGCN), with two variants. Extensive experiments demonstrate TEDGCN's ability of automatically searching for the optimal depth, and it is found that negative-valued depth plays the key role in handling heterophilic graphs. Systematical investigation on the optimal depth is conducted via various real-world and synthetic datasets. It in turn inspires the development of a novel graph augmentation methodology. With clear geometric interpretability, the augmented graph structure possesses supreme advantages over the raw input topology, especially for graphs with heterophily. The main contributions of this paper are summarized as follows:

Analysis. The intrinsic relationship between the negative GCN depth and graph heterophily is discovered; In-depth geometric and spectral explanations are presented.

Problem. Anovel problem of automatic GCN depth tuning for graph homophily/heterophily detection is formulated. To our best knowledge, this work presents the first trial to make GCN's depth negative and trainable by redefining it on the real number domain.

Model. A simple and powerful model TEDGCN with two variants (TEDGCN-S and TEDGCN-D) is proposed. A novel graph augmentation method is discussed.

Experiments. Our model achieves superior performance on semi-supervised node classification tasks on 11 graph datasets.

2 Preliminaries

Notations. We utilize bold uppercase letters for matrices (e.g., A ), bold lowercase letters for column vectors (e.g., u ) and lowercase letters for scalars (e.g., α ). We use the superscript ⊤ for the transpose of matrices and vectors (e.g., A ⊤ and u ⊤ ). An attributed undirected graph G = { A , X } contains an adjacency matrix A ∈ R n × n and an attribute matrix X ∈ R n × q with the number of nodes n and the dimension of node attributes q . D denotes the diagonal degree matrix of A . The adjacency matrix with self-loops is given by ˜ A = A + I ( I is the identity matrix), and all variables derived from ˜ A are decorated with symbol ˜ , e.g., ˜ D represents the diagonal degree matrix of ˜ A . M d stands for the d -th power of matrix M , while the parameter and node embedding matrices in the d -th layer of a GCN are denoted by W ( d ) and H ( d ) .

1

1

1

𝑣

!

𝑣

"

𝑣

#

𝑣

!

𝑣

"

𝑣

#

normalization

1

2

1

2

1

2

=

𝑣

!

𝑣

"

𝑣

#

1 - 𝜆

!

= 1, (𝜆

!

= 0)

1

3

1

3

1

3

1

3

1

3

1

3

𝑣

!

𝑣

"

𝑣

#

-

1

3

1

6

-

1

3

2

3

1

6

1

6

𝑣

!

𝑣

"

𝑣

#

1 - 𝜆

"

= -

1

2

, (𝜆

"

=

3

2

)

-

1

2

1

2

1

2

+

+

𝐀

𝐒 = 𝐃

! "

#

𝐀𝐃

! "

#

eigengraph 1

eigengraph 2

eigengraph 3

1 - 𝜆

#

= -

1

2

, (𝜆

#

=

3

2

)

Graph convolutional network (GCN) and simplified graph convolutional network (SGC). The layer-wise message-passing and aggregation of GCN [28] is given by

where H ( d ) / H ( d +1) stands for the embedding matrix ( H (0) = X ) in the d -th/ ( d +1) -th layer; W ( d ) is the trainable parameter matrix; and σ ( · ) is the non-linear activation function. With σ ( · ) removed in each layer, SGC [59] is obtained as below:

where ˜ S = ˜ D -1 2 ˜ A ˜ D -1 2 , and the parameter of each layer W ( i ) are compressed into one trainable W = ∏ d -1 i =0 W ( i ) .

̸

Graph Laplacian and spectrum. In graph theory, graph Laplacian L = D -A and its symmetrically normalized correspondence L sym = I -D -1 2 AD -1 2 possess critical properties of the underlying graph G . L sym has eigenvalues [ λ 1 , λ 2 , . . . , λ n ] , where λ i ∈ [0 , 2) , ∀ i ∈ { 1 , 2 , . . . , n } [11]. 1 Here, they are put in the ascending order: 0 = λ 1 ≤ λ 2 ≤ · · · ≤ λ n < 2 . It can be eigen-decomposed as: L sym = UΛU ⊤ , where U = [ u 1 , u 2 , . . . , u n ] is the eigenvector matrix ( u i ⊥ u j , ∀ i = j ), and Λ is the diagonal eigenvalue matrix. For each eigenvector u i , we have u i u ⊤ i ∈ R n × n . As we will show in Section 3, this n × n matrix can be viewed as the weighted adjacency matrix of a graph with possible negative edges, which we name as the i -th eigengraph of G . Accordingly, L sym can be written as the linear combination of all eigengraphs weighted by the corresponding eigenvalues [11]:

where the first eigenvalue λ 1 = 0 [46]. Thus, for SGC, we have

A SGC with d layers requires d consecutive graph convolution operations, which involves the multiplication of ˜ S by d times. Due to the orthogonality of ˜ U , namely, ˜ U ⊤ ˜ U = I , we obtain

where 1 -˜ λ i ∈ ( -1 , 1] , and the depth d of SGC serves as the power of ˜ S 's eigenvalues. ˜ S d can be viewed as the sum of eigengraphs ˜ u i ˜ u ⊤ i weighted by coefficients (1 -˜ λ i ) d .

Graph homophily and heterophily. Various homophily/heterophily measures are developed to capture how edges tend to link nodes with the same labels and similar attributes, such as edge homophily[1, 76, 75, 37], node homophily[41], and class homophily[34]. Without the loss of generality, we focus on one mostly used homophily metric, the edge homophily: h ( G ) = ∑ i,j, A [ i,j ]=1 ⟨ y [ i ]= y [ j ] ⟩ ∑ i,j A [ i,j ] ∈ [0 , 1] , where ⟨ x ⟩ = 1 if x is true and 0 otherwise. A graph is more homophilic for h ( G ) closer to 1 or more heterophilic for h ( G ) closer to 0 .

3 Model

Overview. Firstly, we establish the intrinsic relationship between graph spectrum and graph heterophily. Secondly, we show how the positive/negative depth d of a GCN affects the eigengraph weights which in turn impacts the algorithm's capability to capture homophilic/heterophilic graph signals. Thirdly, with the help of functional calculus [45], we present the theoretical feasibility of extending the domain of d from N + to R . Finally, by making d a trainable parameter, we present our model TEDGCN and its variants, which are capable of automatically detecting the homophily/heterophily of the input graph and finding the optimal depth.

̸

Eigenvector frequency and entry deviation. The frequency of graph Laplacian eigenvector u i reflects how much the j -th entry u i [ j ] deviates from the k -th entry u i [ k ] for each connected node pair v j and v k in G [46]. This deviation is measured by the set of zero crossings of u i : Z ( u i ) := { e = ( v j , v k ) ∈ E : u i [ j ] u i [ k ] < 0 , j = k } , where E is the set of edges in graph G . Larger/smaller |Z ( u i ) | indicates higher/lower eigenvector frequency. A zero-crossing also corresponds a negative weighted edge in an eigengraph. Due to the widely existing positive correlation between λ i and |Z ( u i ) | [46], large/small eigenvalues mostly correspond to the high/low frequencies of the related eigenvectors. As illustrated by the toy example of n = 3 in Figure 1, for λ 1 = 0 , we have |Z ( u 1 ) | = 0 , and eigengraph u 1 u ⊤ 1 is fully-connected; negative edge weights exist in the 2nd and 3rd eigengraphs, indicating more zero crossings ( |Z ( u 2 ) | = 1 and |Z ( u 3 ) | = 2 ) and higher eigenvector frequencies.

̸

Entry deviation and graph heterophily. Reflected by entry deviations, eigenvector frequency and zero-crossings determine the number of negatively weighted edges ( u i [ j ] u i [ k ] < 0 , j = k ) in a corresponding eigengraph u i u ⊤ i . For a GCN/SGC, conducting message-passing along negatively weighted edges makes the embeddings of the connected node pairs ( v j , v k ) dissimilar. Naturally, higher embedding dissimilarity is more likely to render different label predictions between the node pair, and this generates heterophily. Thus, eigenvectors with higher frequencies and accordingly higher entry deviations contribute more to heterophilic graph signals, and vice versa.

Graph heterophily and GCN's depth. Based on above analyses, high frequency eigenvectors and their corresponding eigengraphs have advantage on capturing graph heterophily. A graph neural network should be able to give high frequency eigengraphs larger weights when modeling heterophilic graphs, whereas low frequency ones should carry larger weights when dealing with homophilic graphs. The weights of eigengraphs are controlled by GCN/SGC's depth d as follows. For a SGC of depth d , the weight of the i -th eigengraph is (1 -˜ λ i ) d , and changing the depth d of SGC adjusts the weights of different eigengraphs. Therefore, depth d controls the model's capability of effectively filtering low/high-frequency signals for graph homophily/heterophily.

Redefined GCN's depth. A question is naturally raised: instead of manually setting the depth d , can d be built into the model as a trainable parameter? If yes, a proper set of the eigengraph weights matching the graph homophily/heterophily can be automatically reached by finding the optimal d in an end-to-end fashion during training.

Differentiable variables need continuity, which requires the extension of depth d from the discrete positive integer domain ( N + ) to the continuous real number domain R . According to functional calculus [45], applying an arbitrary matrix function f on a graph Laplacian L sym is equivalent to applying the same function only on the eigenvalue matrix Λ :

which also applies to ˜ L sym and ˜ S . Armed with this, we seek to realize an arbitrary depth SGC via a power function as f ( ˜ S ) = ˜ S d = ˜ U ( I -˜ Λ ) d ˜ U ⊤ = ∑ n i =1 (1 -˜ λ i ) d ˜ u i ˜ u ⊤ i ( d ∈ R ) . However, since ˜ λ i ∈ [0 , 2) , we have (1 -˜ λ i ) ≤ 0 when 1 ≤ ˜ λ i < 2 , and for (1 -˜ λ i ) taking zero or negative values, (1 -˜ λ i ) d is not well-defined or involving complex-number-based calculations for a real-valued d (e.g., ( -0 . 5) 3 8 ) [45]. Moreover, even for integer-valued d s under which (1 -˜ λ i ) d is easy to compute, the behavior of (1 -˜ λ i ) d is complicated versus ˜ λ i and diverges when ˜ λ i = 1 for negative d s, as shown in Figure 2a. Thus, the favored weight distribution may be hard to obtain by tuning d .

To avoid such complications and alleviate the difficulties for manipulating the eigengraph weights, a transformation function g ( · ) operating on the graph Laplacian L sym or ˜ L sym is in need to shift g ( λ i ) or g ( ˜ λ i ) into a proper value range so that its power of a real-valued d is easy to obtain and well-behaved versus λ i or ˜ λ i . Without the loss of generality, our following analysis focuses on L sym and λ i . There may exist multiple choices for g ( · ) satisfying the requirements. In this work, we focus on the following transformation function:

This choice of g ( · ) holds three properties: (1) Positive eigenvalues. Since we have L sym 's i -th eigenvalue λ i ∈ [0 , 2) , the corresponding eigenvalue of ˆ S is g ( λ i ) = 1 2 (2 -λ i ) ∈ (0 , 1] . Thus, the d -th power of g ( λ i ) is computable for any d ∈ R . (2) Monotonicity versus eigenvalues λ . As shown in Figure 2b, g ( λ i ) d = (1 -1 2 λ ) d is monotonically increasing/decreasing when λ varies between 0 and 2 under negative/positive depth. (3) Geometric interpretability. Filter ˆ S can be expressed as:

0.0

0.5

1.0

1.5

4

3

2

1

0

1

2

3

4

(1

)

d

d

=2

d

=1

d

=0

d

=

1

d

=

2

(a)

(1

-

˜

λ

)

d

.

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

(1

0.5

)

d

d

=2

d

=1

d

=0

d

=

1

d

=

2

(b)

(1

-

1

λ

)

d

.

2

As shown in Figure 3, in the spatial domain, ˆ S is obtained via 3 operations on adjacency matrix A : normalization, adding self-loops, and scaling all edge weights by 1 2 (a type of lazy random walk [39]), while ˜ S in vanilla GCN/SGC contains 2 operations: adding self-loops and normalization.

With the help of transformation g , the depth d is redefined on real number domain, and the message propagation process of depth d can be realized via the following steps: (1) Eigen-decompose L sym ; (2) Calculate ˆ S d via weight g ( λ i ) d and the weighted sum of all eigengraphs: ˆ S d = U ˆ Λ d U ⊤ = ∑ n i =1 g ( λ i ) d u i u ⊤ i (3) Multiply ˆ S d with original node attributes X .

Negative depth explained. First, an intuitive explanation of negative d can be obtained from the perspective of matrix inverse and message diffusion process when d takes integer values. Since ˆ S -1 ˆ S = U ˆ Λ -1 U ⊤ U ˆ Λ 1 U ⊤ = I , ˆ S -1 is the inverse matrix of ˆ S . In diffusion dynamics, X can be viewed as an intermediate state generated in a series of message propagation steps. ˆ SX effectively propagates the message one-step forward, while ˆ S -1 can cancel the effect of ˆ S on X and recover the original message by moving backward: ˆ S -1 ˆ SX = X . Accordingly, ˆ S -1 X traces back to the message's previous state in

𝑣

!

𝑣

"

𝑣

#

𝑣

!

𝑣

"

𝑣

#

1

4

1

4

1

4

1

2

1

2

1

2

𝑣

!

𝑣

"

𝑣

#

1

1

1

1

2 𝐃

! "

#

𝐀𝐃

! "

#

1

2

𝐈

!

𝐒

𝑣

!

𝑣

"

𝑣

#

1

3

1

3

1

3

1

3

1

3

1

3

#

𝐒

1

1

1

the series. Non-integer d indicates the back- or forward propagation can be a continuous process. Second, as the depth becomes negative, it primarily keeps the eigengraphs of high frequency (See Figure 2b), which have more negative edges. Therefore, it is equivalent to turning some positive edges in the original graph into negative edges in the augmented graph, which in turn makes node embeddings dissimilar in the message-passing process and thus naturally handles the heterophilic graphs (See Figure 1). More discussions on the impact of negative depth in spatial domain are presented in Section 4.4.

TEDGCN-S algorithm. By further making d a trainable parameter, we present our model, Trainable Depth-GCN-Single (TEDGCN-S), whose final node embedding matrix is given by

where σ ( · ) is the nonlinear activation function; W is a trainable parameter matrix; and d is the trainable depth parameter. As observed from Figure 2b, weight distribution of different frequencies/eigengraphs is tuned via d : (1) for d = 0 , the weight is uniformly distributed among all frequency components ( g ( λ i ) d = 1 ), which implies that no particular frequency is preferred by the graph signal; (2) for d > 0 , weight g ( λ i ) d decreases with the corresponding frequency, which indicates the low frequency components are favored so that TEDGCN-S effectively functions as a low-pass filter and therefore captures graph homophily; (3) for d < 0 , high frequency components gains amplified weights so that TEDGCN-S serves as a high-pass filter capturing graph heterophily. During training, TEDGCN-S tunes its frequency filtering functionality to suit the underlying graph signal by automatically finding the optimal d .

TEDGCN-D algorithm. During optimization, TEDGCN-S embraces a single depth d unified for all eigengraphs and selects its preferences for either homophily or heterophily. However, TEDGCN-S requires a full eigen-decomposition of L sym , which can be expensive for large graphs. Additionally, the high and low frequency components in a graph signal may not be mutually exclusive, namely, there exists the possibility for a graph to simultaneously possess homophilic and heterophilic counterparts. Therefore, we propose the second variant of TEDGCN: TEDGCN-D (Dual), which introduces two separate trainable depths, d h and d l , to gain more flexible weighting of the high and low frequency related eigengraphs respectively. Arnoldi method [30] is adopted to conduct EVD on L sym and obtain the topK largest and smallest eigen-pairs ( λ i , u i ) s. By denoting U l = U [: , 0 : K ] and U h = U [: , n -K : n ] ( U l and U h ∈ R n × K ), we define a new diffusion matrix ˆ S dual ( d l , d h , K ) as

where ˆ Λ l ∈ R K × K and ˆ Λ h ∈ R K × K are diagonal matrices of the topK smallest and largest eigenvalues. 2 The final node embedding of TEDGCN-D is presented as

where depths d l and d h are trainable; and W is a trainable parameter matrix. We make TEDGCN-D scalable on large graphs by choosing K ≪ n , so that ˆ S dual ( d l , d h , K ) approximates the full diffusion matrix by covering only a small subset of all eigengraphs. For small graphs, we use K = ⌊ n 2 ⌋ to include all eigengraphs, and ˆ S dual ( d l , d h , K ) thus gains higher flexibility than ˆ S with the help of the two separate depth parameters instead of a unified one.

Complexity Analysis. TEDGCN has two major steps: (1) conduct eigen-decomposition of ˆ A ; (2) train a one-layer GCN on the augmented graph ˆ S d . For Step (1), naive eigen-decomposition has a time complexity of O( n 3 ). However, by adopting the Lanczos method [30] and only keeping the topK largest/smallest eigenvalues in TEDGCN-D, the time complexity can be reduced to O ( nK 2 + mK ) [24], where m is the number of edges and K << n . For Step (2), via dense matrix multiplications, computing the approximated ˆ S d takes O( nK 2 + n 2 K ), and conducting message propagation takes O( n 2 q + nqc ), where q and c denote the dimension of node feature vectors and the number of node classes. Potential improvements of efficiency: since the diffusion matrix in Subsection 4.4 of the augmented graph is sparse for known real-world graphs, the complexity of Step (2) can also be further simplified: the O( n 2 K ) term can be reduced to O( | E aug | K ), where E aug is the edge set of the augmented graph.

Differences with ODE-based GNNs . Previous attempts on GNNs with continuous diffusion are mostly inspired by graph diffusion equation, an Ordinary Difference Equation (ODE) characterizing the dynamical message propagation process versus time. In contrast, our framework starts from discrete graph convolution operations without explicitly involving ODE. CGNN [60] aims to build a deep GNN immune to over-smoothing by adopting the neural ODE framework [9]. But its time parameter t is a non-trainable hyper-parameter predefined within the positive domain, which is the key difference with TEDGCN. A critical CGNN component for preventing over-smoothing, restart distribution (the skip connection from the first layer), is not needed in our framework. Moreover, CGNN applies the same depth to all frequency components, while TEDGCN-D has the flexibility to adopt two different depths respectively to be adaptive to high and low frequency components. GRAND [8] introduces non-Eular multi-step schemes with adaptive step size to obtain more precise solutions of the diffusion equation. Its depth (i.e., the total integration time) is continuous but still

predefined/non-trainable and only admits positive values. DGC [58] decouples the SGC depth into two predefined non-trainable hyper-parameters: a positive real-valued T controlling the total time and a positive integer-valued K dgc corresponding to the number of diffusion steps. However, realizing negative depth in DGC is non-applicable since the implementation is through propagation by K dgc times, rather than through an arbitrary real-valued exponent d on eigengraph weights in TEDGCN.

4 Experiment

In this section, we evaluate the proposed TEDGCN on the semi-supervised node classification task on both homophilic and heterophilic graphs.

4.1 Experiment Setup

Datasets. We use 11 datasets for evaluation, including 4 homophilic graphs: Cora [28], Citeseer [28], Pubmed [28] and DBLP [6], and 7 heterophilic graphs: Cornell [41], Texas [41], Wisconsin [41], Actor [41], Chameleon [44], Squirrel [44], and cornell5 [15]. We collect all datasets from the public GCN platform Pytorch-Geometric [15]. For Cora, Citeseer, Pubmed with data splits in Pytorch-Geometric, we keep the same training/validation/testing set split as in GCN [28]. For the remaining 8 datasets, we randomly split every dataset into 20/20/60% for training, validation, and testing.

Baselines and metrics. Here, we compare our model with 7 baseline methods, including 4 classic GNNs: GCN [28], SGC [59], APPNP [29] and ChebNet [12], and 3 GNNs tailored for heterophilic graphs: FAGCN [5], GPRGNN [10] and H2GCN [76]. Accuracy (ACC) is used as the evaluation metric. We report the average ACCs with the standard deviation (std) for all methods, each obtained by 5 runs with different initializations.

Additional contents. In Appendix, we provide more contents related to experiments including (1) the statistics of all datasets (Appendix A.1); (2) experimental results on synthetic graphs with controllable edge homophily/heterophily levels to demonstrate the effectiveness of negative depth (Appendix A.2); (3) comparison with additional methods including ACM-GCN [37, 38], LINKX [34], BernNet [22], GBK-GNN [14] , HOG-GCN [57], Geom-GCN [41] and GloGNN [33] under the commonly used fully-supervised setting that training/validation/testing set split is 48/32/20% for training, validation, and testing based on the results reported in the original papers (Appendix A.3); (4) implementation details (Appendix A.4); and (5) visualization of node embeddings w.r.t. d (Appendix A.5).

4.2 Node Classification

The semi-supervised node classification performances on homophilic graphs and heterophilic graphs are shown in Table 1 and Table 2 respectively.

Homophilic graphs . From Table 1, it is observed that different methods have similar performance on homophilic graphs. TEDGCN-S achieves the best accuracies on two datasets: Cora and DBLP. On the remaining two datasets, TEDGCN-S is only 1 . 1% and 0 . 4% below the best baselines (APPNP on Citeseer and SGC on Pubmed). For TEDGCN-D, it obtains similar performance as the other methods, even though it only uses the topK largest/smallest eigen-pairs.

Heterophilic graphs . Our TEDGCN-S or TEDGCN-D outperforms every baseline on all heterophilic graphs, as shown in Table 2. These results demonstrate that without manually setting the

0

1

2

3

4

5

d

50

55

60

65

70

75

80

85

90

Accuracy(%)

the optimal (5.029, 82.8%)

Performance on Cora

0

1

2

3

4

5

d

50

55

60

65

70

75

80

Accuracy(%)

the optimal (3.735, 71.4%)

Performance on Citeseer

(b) Citeseer

1

0

1

d

20

25

30

35

40

45

Accuracy(%)

the optimal (-0.027, 36.9%)

Performance on Actor

(c) Actor

3.9

3.8

3.7

3.6

3.5

3.4

3.3

d

20

25

30

35

40

45

50

Accuracy(%)

the optimal (-3.751,41.5%)

model depth and without the prior knowledge of the input graph (i.e., whether the input graph is homophilic or heterophilic), TEDGCN has the capability of automatically detecting the underlying graph heterophily. We have an interesting observation: on 3 large datasets, Squirrel, Chameleon, and cornell5, even with only a small portion of the eigengraphs, TEDGCN-D is able to achieve better performance than TEDGCN-S with the complete set of eigengraphs. This suggests that the graph signal in some real-world graphs might be dominated by a few low and high frequency components, and allowing two independent depth parameters in TEDGCN-D brings the flexibility to capture the low and high frequencies at the same time.

4.3 Trainable Depth

A systematic study is conducted on the node classification performance w.r.t. the trainable depth d . Optimal depth . In Figure 4, the optimal depths and their corresponding classification accuracies are annotated. For two homophilic graphs, Cora and Citeseer, the optimal depths are positive ( 5 . 029 and 3 . 735 ) in terms of the best ACCs, while for two heterophilic graphs, Actor and Squirrel, the optimal depths are negative ( -0 . 027 and -3 . 751 ). These results demonstrate that our model can indeed automatically capture graph heterophily/homophily by finding the suitable depth to suppress or amplify the relative weights of the corresponding frequency components. Namely, high/low frequency components are suppressed for homophilic/heterophilic graphs respectively.

Close to zero depth. For the two homophilic graphs in Figures 4a and 4b, sharp performance drop is observed when depth d approaches 0, since the eigengraphs gain close-touniform weights. For the heterophilic Actor dataset, its optimal depth -0 . 027 is close to 0 , as shown in Figure 4c. In addition, the performance of TEDGCN-D ( 27 . 6% ) is similar to that of GCN ( 27 . 5% ), both of which are much worse than TEDGCN-S ( 35 . 3% ). This result indicates that Actor is a special graph where all frequency compo-

nents have similar importance. Due to the absence of the intermediate frequency components between the high- and low-end ones, the performance of TEDGCN-D is severely impacted. For vanilla GCN, the suppressed weights of the high frequency components deviate from the near-uniform spectrum and thus lead to low ACC on this dataset.

4.4 Graph Augmentation and Geometric Insights

It is especially interesting to analyze what change a negative depth brings to the spatial domain and how such change impacts the subsequent model performance.

Graph augmentation via negative depth. By picking the optimal depth d according to the best performance on the validation set, a new diffusion matrix ˆ S d is obtained. With the optimal d fixed,

)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

0.0

0.5

1.0

1.5

2.0

2.5

Weights

S

d

S

(a) Texas (

d

=

-

0

.

328

)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Weights

S

d

S

(b) Cornell (

d

=

-

0

.

362

)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

0.0

0.5

1.0

1.5

2.0

2.5

Weights

S

d

S

(c) Wisconsin (

d

=

-

0

.

280

(a) Texas

(b) Cornell

(c) Wisconsin

0

25

1.00

0.75

50

0.50

75

0.25

100

0.00

125

-0.

150

-0.

175

-0.

0

25

50

75

100

125

150

175

0

25

1.0

50

0.5

75

100

0.0

125

0.

150

175

1.

0

25

50

75

100

125

150

175

0

0.8

50

- 0.6

- 0.4

100

0.2

150

0.0

'0-

200

-0.

0.

250

0

50

100

150

200

250

substituting the normalized adjacency matrix ˜ D -1 2 ˜ A ˜ D -1 2 in Eq. (1) by ˆ S d is equivalent to applying the vanilla GCN to a new topology. This topology effectively plays the role of a structural augmentation for the original graph. The impact of such augmentation on performance is tested on 3 heterophilic graphs: Texas, Cornell and Wisconsin, as shown in Table 3. Apparently, for the vanilla GCN, the performance obtained with this new topology is superior over that with the raw input graph: it dramatically brings 20% -30% lifts in ACC. Moreover, the augmented topologies also make vanilla GCN outperform TEDGCN-S and TEDGCN-D on 2 out of the 3 datasets. By nature, the augmented graph is a re-weighted linear combination of the eigengraphs, and its topological structure intrinsically assigns higher weights to eigengraphs corresponding to higher frequencies, as shown in Figure 5.

Geometric properties. To further understand how the topology of ˆ S d with a negative optimal d differs from that of ˆ S and why the performance is significantly boosted, heat maps of ( ˆ S d -ˆ S ) s are presented in Figure 6. First, the dark red diagonal line in the heat map indicates the weights of self-loops are significantly strengthened in the augmented graph, and as a result, in consistency with the previous findings [71], the raw node attributes make more contributions in predicting their labels. These strengthened self-weights also play the similar role as restart distribution or skip connections [60] preventing the node embeddings becoming over-smoothed. In addition, there is a horizontal line and a vertical line (light yellow line marked by dashed ovals) in each heat map in Figure 6, correspond to the hub node in the graph, namely the node with the largest degree. Interestingly, the connections between this node and most other nodes in the graph experience a negative weight change. Therefore, the influence of the hub node on most other nodes are systematically reduced. Consequently, the augmentation amplifies the deviations between node embeddings and facilitates the characterization of graph heterophily.

5 Related Works

Graph convolutional network (GCN). GCN models can be mainly divided into two categories: (1) spectral graph convolutional networks and (2) spatial convolutional networks. In (1), Spectral CNN [7] borrows the idea from convolutional neural network [18] to construct a diagonal matrix as the convolution kernel. ChebNet [12] adopts a polynomial approximation of the convolution kernel. GCN [28] further simplifies the ChebNet via the first order approximation. Recently, [22, 4, 57] propose more advanced filters as the convolution kernel. Most works in (2) follow the message-passing mechanism. GraphSAGE [19] iteratively aggregates features from local neighborhood. GAT [50] applies self-attention to the neighbors. APPNP [29] deploys personalized pagerank [49] to sample nodes for aggregration. MoNet [40] unifies GCNs in the spatial domain.

The depth of GCN and over-smoothing. A large amount of works focus on the over-smoothing issue. Its intrinsic cause is demystified: a linear GCN layer is a Laplacian smoothing operator [32, 59]. PairNorm [69] forces distant nodes to be distinctive by adding an intermediate normalization layer. Dropedge [43], DeepGCN [31], AS-GCN [23], and JK-net [61] borrow the idea of ResNet [21] to dis-intensify smoothing. DeeperGXX [70] adopts a topology-guided graph contrastive loss for connected node pairs to obtain discriminative representations. Most works aim to build deep GCNs (i.e., d is a large positive integer) by reducing over-smoothing, while TEDGCN extends the depth from N + to R and explores the negative depth.

Node classification on homophilic and heterophilic graphs. GCN/GNN models mostly follow the homophily assumption that connected nodes tend to share similar labels [28, 50, 19]. Recently, heterophilic graphs, in which neighbors often have disparate labels, attract lots of attention. GeomGCN [41] and H2GCN [76] extend the neighborhood for aggregation. FAGCN [5] and GPRGNN [10] adaptively integrate the high/low frequency signals with trainable parameters. Alternative message-passing mechanisms have been proposed in HOG-GCN [57] and CPGNN [75]. The latest related works include ACM-GCN [37, 38], LINKX [34], BernNet [22], GloGNN [33] and GBK-GNN [14]. Other works can be found in a recent survey [72].

6 Conclusion and Future Work

To our best knowledge, this work presents the first effort to make GCN's depth become negative and trainable by redefining it on the real number domain. We unveil the intrinsic connection between negative GCN depth and graph heterophily. A novel problem of automatic GCN depth tuning for graph homophily/heterophily detection is formulated, and we propose a simple and powerful solution named TEDGCN with two variants (TEDGCN-S and TEDGCN-D). An effective graph augmentation method is also discussed via the new understanding on the message propagation mechanism generated by the negative depth. Superior performance of our method is demonstrated via extensive experiments with semi-supervised node classification on 11 graph datasets. The new insights on GCN's depth obtained by our work may open a new direction for future research on spectral and spatial GNNs. Our paper studies the depth of graph convolutional network and graph spectrum, which has no negative ethical impacts on society. Since TEDGCN requires to conduct eigen-decomposition of the graph Laplacian, it is not directly applicable to inductive and dynamic graph learning problems. In addition, there may exist other options for the transformation functions such as the pseudoinverse of the Laplacian. We leave these for future exploration.

References

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In Proceedings of the 36 th International Conference on Machine Learning , 2019.

Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. Ee-net: Exploitation-exploration neural networks in contextual bandits. arXiv preprint arXiv:2110.03177 , 2021.

Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. Neural exploitation and exploration of contextual bandits. arXiv preprint arXiv:2305.03784 , 2023.

Filippo Maria Bianchi, Daniele Grattarola, Lorenzo Livi, and Cesare Alippi. Graph neural networks with convolutional arma filters. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 3950-3957, 2021.

Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. arXiv preprint arXiv:1707.03815 , 2017.

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 , 2013.

Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. In International Conference on Machine Learning , pages 1407-1418. PMLR, 2021.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. arXiv preprint arXiv:2006.07988 , 2020.

Fan RK Chung and Fan Chung Graham. Spectral graph theory , volume 92. American Mathematical Soc., 1997.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems , 29, 2016.

Boxin Du, Si Zhang, Yuchen Yan, and Hanghang Tong. New frontiers of multi-network mining: Recent developments and future trend. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 4038-4039, 2021.

Lun Du, Xiaozhou Shi, Qiang Fu, Xiaojun Ma, Hengyu Liu, Shi Han, and Dongmei Zhang. Gbk-gnn: Gated bi-kernel graph neural networks for modeling both homophily and heterophily. In Proceedings of the ACM Web Conference 2022 , pages 1550-1558, 2022.

Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. IEEE Signal Processing Magazine , 30:83-98, 2019.

Dongqi Fu, Liri Fang, Ross Maciejewski, Vetle I Torvik, and Jingrui He. Meta-learned metrics over multi-evolution temporal graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 367-377, 2022.

Dongqi Fu, Zhe Xu, Bo Li, Hanghang Tong, and Jingrui He. A view-adversarial framework for multi-view network embedding. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 2025-2028, 2020.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, 2016.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. In International Conference on Machine Learning , pages 8230-8248. PMLR, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Mingguo He, Zhewei Wei, Hongteng Xu, et al. Bernnet: Learning arbitrary graph spectral filters via bernstein approximation. Advances in Neural Information Processing Systems , 34:14239-14251, 2021.

Wenbing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. Adaptive sampling towards fast graph representation learning. Advances in neural information processing systems , 31, 2018.

Zexi Huang, Arlei Silva, and Ambuj Singh. A broader picture of random-walk based graph embedding. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 685-695, 2021.

Baoyu Jing, Hanghang Tong, and Yada Zhu. Network of tensor time series. In Proceedings of the Web Conference 2021 , pages 2425-2437, 2021.

Baoyu Jing, Yuchen Yan, Kaize Ding, Chanyoung Park, Yada Zhu, Huan Liu, and Hanghang Tong. Sterling: Synergistic representation learning on bipartite graphs. arXiv preprint arXiv:2302.05428 , 2023.

Baoyu Jing, Yuchen Yan, Yada Zhu, and Hanghang Tong. Coin: Co-cluster infomax for bipartite graphs. arXiv preprint arXiv:2206.00006 , 2022.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997 , 2018.

Richard B Lehoucq, Danny C Sorensen, and Chao Yang. ARPACK users' guide: solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods . SIAM, 1998.

Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgcns: Can gcns go as deep as cnns? In Proceedings of the IEEE/CVF international conference on computer vision , pages 9267-9276, 2019.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference on artificial intelligence , 2018.

Xiang Li, Renyu Zhu, Yao Cheng, Caihua Shan, Siqiang Luo, Dongsheng Li, and Weining Qian. Finding global homophily in graph neural networks when meeting heterophily. arXiv preprint arXiv:2205.07308 , 2022.

Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. Advances in Neural Information Processing Systems , 34:20887-20902, 2021.

Hongyi Ling, Zhimeng Jiang, Meng Liu, Shuiwang Ji, and Na Zou. Graph mixup with soft alignments. arXiv preprint arXiv:2306.06788 , 2023.

Lihui Liu, Boxin Du, Jiejun Xu, Yinglong Xia, and Hanghang Tong. Joint knowledge graph completion and question answering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1098-1108, 2022.

Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. Is heterophily a real nightmare for graph neural networks to do node classification? arXiv preprint arXiv:2109.05641 , 2021.

Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. Revisiting heterophily for graph neural networks. arXiv preprint arXiv:2210.07606 , 2022.

Sitao Luan, Mingde Zhao, Chenqing Hua, Xiao-Wen Chang, and Doina Precup. Complete the missing half: Augmenting aggregation filtering with diversification for graph convolutional networks. arXiv preprint arXiv:2008.08844 , 2020.

Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5115-5124, 2017.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. arXiv preprint arXiv:2002.05287 , 2020.

Shane Roach, Connie Ni, Alexei Kopylov, Tsai-Ching Lu, Jiejun Xu, Si Zhang, Boxin Du, Dawei Zhou, Jun Wu, Lihui Liu, et al. Canon: Complex analytics of network of networks for modeling adversarial activities. In 2020 IEEE International Conference on Big Data (Big Data) , pages 1634-1643. IEEE, 2020.

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. arXiv preprint arXiv:1907.10903 , 2019.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks , 9(2):cnab014, 2021.

Kamal Shah and Baver Okutmu¸ stur. Functional Calculus . BoD-Books on Demand, 2020.

David I Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE signal processing magazine , 30(3):83-98, 2013.

Yuxin Tang, Zhimin Ding, Dimitrije Jankov, Binhang Yuan, Daniel Bourgeois, and Chris Jermaine. Auto-differentiation of relational computations for very large scale machine learning. arXiv preprint arXiv:2306.00088 , 2023.

Gabriel Taubin. A signal processing approach to fair surface design. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques , pages 351-358, 1995.

Hanghang Tong, Christos Faloutsos, and Jia-Yu Pan. Fast random walk with restart and its applications. In Sixth international conference on data mining (ICDM'06) , pages 613-622. IEEE, 2006.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Dingsu Wang, Yuchen Yan, Ruizhong Qiu, Yada Zhu, Kaiyu Guan, Andrew Margenot, and Hanghang Tong. Networked time series imputation via position-aware graph enhanced variational autoencoders. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2256-2268, 2023.

Ruijie Wang, Baoyu Li, Yichen Lu, Dachun Sun, Jinning Li, Yuchen Yan, Shengzhong Liu, Hanghang Tong, and Tarek F Abdelzaher. Noisy positive-unlabeled learning with self-training for speculative knowledge graph reasoning. arXiv preprint arXiv:2306.07512 , 2023.

Ruijie Wang, Zheng Li, Dachun Sun, Shengzhong Liu, Jinning Li, Bing Yin, and Tarek Abdelzaher. Learning to sample and aggregate: Few-shot reasoning over temporal knowledge graphs. Advances in Neural Information Processing Systems , 35:16863-16876, 2022.

Ruijie Wang, Zheng Li, Jingfeng Yang, Tianyu Cao, Chao Zhang, Bing Yin, and Tarek Abdelzaher. Mutually-paced knowledge distillation for cross-lingual temporal knowledge graph reasoning. In Proceedings of the ACM Web Conference 2023 , pages 2621-2632, 2023.

Ruijie Wang, Zheng Li, Danqing Zhang, Qingyu Yin, Tong Zhao, Bing Yin, and Tarek Abdelzaher. Rete: retrieval-enhanced temporal event forecasting on unified query product evolutionary graph. In Proceedings of the ACM Web Conference 2022 , pages 462-472, 2022.

Ruijie Wang, Yuchen Yan, Jialu Wang, Yuting Jia, Ye Zhang, Weinan Zhang, and Xinbing Wang. Acekg: A large-scale knowledge graph for academic data mining. In Proceedings of the 27th ACM international conference on information and knowledge management , pages 1487-1490, 2018.

Xiyuan Wang and Muhan Zhang. How powerful are spectral graph neural networks. arXiv preprint arXiv:2205.11172 , 2022.

Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Dissecting the diffusion process in linear graph convolutional networks. Advances in Neural Information Processing Systems , 34:5758-5769, 2021.

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning , pages 6861-6871. PMLR, 2019.

Louis-Pascal Xhonneux, Meng Qu, and Jian Tang. Continuous graph neural networks. In International Conference on Machine Learning , pages 10432-10441. PMLR, 2020.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning , pages 5453-5462. PMLR, 2018.

Zhe Xu, Yuzhong Chen, Qinghai Zhou, Yuhang Wu, Menghai Pan, Hao Yang, and Hanghang Tong. Node classification beyond homophily: Towards a general solution. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2862-2873, 2023.

Yuchen Yan, Lihui Liu, Yikun Ban, Baoyu Jing, and Hanghang Tong. Dynamic knowledge graph alignment. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 4564-4572, 2021.

Yuchen Yan, Si Zhang, and Hanghang Tong. Bright: A bridging algorithm for network alignment. In Proceedings of the Web Conference 2021 , pages 3907-3917, 2021.

Yuchen Yan, Qinghai Zhou, Jinning Li, Tarek Abdelzaher, and Hanghang Tong. Dissecting cross-layer dependency inference on multi-layered inter-dependent networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 2341-2351, 2022.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

Zhichen Zeng, Boxin Du, Si Zhang, Yinglong Xia, Zhining Liu, and Hanghang Tong. Hierarchical multi-marginal optimal transport for network alignment. arXiv preprint arXiv:2310.04470 , 2023.

Zhichen Zeng, Si Zhang, Yinglong Xia, and Hanghang Tong. Parrot: Position-aware regularized optimal transport for network alignment. In Proceedings of the ACM Web Conference 2023 , pages 372-382, 2023.

Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In International Conference on Learning Representations , 2019.

Lecheng Zheng, Dongqi Fu, and Jingrui He. Tackling oversmoothing of gnns with contrastive learning. arXiv preprint arXiv:2110.13798 , 2021.

Wenqing Zheng, W Edward Huang, Nikhil Rao, Sumeet Katariya, Zhangyang Wang, and Karthik Subbian. Cold brew: Distilling graph node represen- tations with incomplete or missing neighbor- hoods. In International Conference on Learning Representations , 2022.

Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S Yu. Graph neural networks for graphs with heterophily: A survey. arXiv preprint arXiv:2202.07082 , 2022.

Qinghai Zhou, Liangyue Li, Nan Cao, Lei Ying, and Hanghang Tong. Admiring: Adversarial multi-network mining. In 2019 IEEE International Conference on Data Mining (ICDM) , pages 1522-1527. IEEE, 2019.

Qinghai Zhou, Liangyue Li, Xintao Wu, Nan Cao, Lei Ying, and Hanghang Tong. Attent: Active attributed network alignment. In Proceedings of the Web Conference 2021 , pages 3896-3906, 2021.

Jiong Zhu, Ryan A Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K Ahmed, and Danai Koutra. Graph neural networks with heterophily. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11168-11176, 2021.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems , 33:7793-7804, 2020.

A Appendix

The appendix is structured as follows:

Subsection A.1 presents the statistics of datasets used in the Experiment Section (Section 4) in the main contents.

Subsection A.2 presents the experimental results on synthetic datasets with controllable homophily/heterophily level.

Subsection A.3 compares TEDGCN with some latest methods on a different split (48%, 32%, 20%) for training, validation and testing of benchmark datasets. In this subsection, we also compare all methods (15 baselines and TEDGCN) under a unified setting on Texas, Cornell, Wisconsin and Actor datasets.

Subsection A.4 introduces the details of the implementation of TEDGCN.

Subsection A.5 is a visualization of node embeddings w.r.t. different depths ( d values) on both homophilic (Cora) and heterophilic (Texas) graphs.

A.1 The Statistics of Datasets

A.2 Experiments on Synthetic Datasets

To better demonstrate the relation between the negative depth d and edge homophily/heterphily, we construct synthetic datasets with different homophily levels. The synthetic graphs are generated via scheme adopted in ACM-GCN [38]. Specifically, there are 5 classes, each with 400 nodes. For a fixed homophily level h ( G ) (e.g., h ( G ) = 0 . 1 ), the node degree δ v is sampled for each node via multinomial distribution. With the sampled δ v , h ( G ) δ v intra-class edges and (1 -h ( G )) δ v inter-class edges are randomly generated. We compare TEDGCN with a two-layer SGC. The results on the synthetic datasets with different homophily levels are presented in Table 5. We can observe that (1) as the homophily level h ( G ) decreases, the optimal d turns from positive to negative; and (2) under low homophily levels, the performance of TEDGCN is significantly better than SGC.

A.3 More Baselines and Unified Setting

Under fully-supervised setting, we compare TEDGCN with ACM-GCN, LINKX, BernNet and GloGNN with the standard split of 48%/32%/20%. We directly use the performances of ACM-GCN, LINKX and GloGNN reported in the ACM-GCN paper. For BernNet, we obtain its performance by running its open-sourced code. The results are shown in Table 6. It is observed that the latest

methods, such as ACM-GCN and GloGNN do perform strongly. However, our model achieves the best classification accuracy on 7 out of the 9 datasets, which demonstrates the effectiveness of TEDGCN.

Wealso compare all methods (15 baselines and TeDGCN) under a unified setting. Since some methods do not report their performances on some datasets in their original paper and have not made their code publicly available, we compare all methods' performances on 4 commonly used heterophilic graph datasets: Texas, Cornell, Wisconsin and Actor, under the same public 48%/32%/20% split for training, validation and testing.

From Table 7, we can observe that TeDGCN-S/TeDGCN-D outperforms all the remaining baselines.

A.4 Implementation Details

We set the learning rate to 0 . 005 or 0 . 01 , the decaying weight for the learning rate to 5 e -4 , the number of training epochs to 500. The K largest/smallest eigenvalues for TEDGCN-D is max { 1000 , 0 . 1 n } , where n is the number of nodes in the input graph. For all baselines, the parameters are set according to their original papers. All experiments are run on a Tesla-V100 GPU. Since this work is conducted during internship in Visa Research and the code can not be made public according to the policy of Visa Research. If you have any questions about the implementation, please email yucheny5@illinois.edu to get some help.

A.5 Visualization of Node Embeddings w.r.t. Negative Depth d

We conduct a visualization of node embedding generated by TEDGCN under different depths for homophilic (Cora) and heterophilic (Texas) graphs. As shown in Figure 7, for Cora, when d = 5 , the nodes in different classes form clearly distinguishable clusters (ACC=82.4%). When the depth decreases from a positive value d = 5 (ACC=82.4%) to a negative value d = -2 (ACC=29.5%), the nodes belonging to different classes/colors mix with each other, and the clearly clustered structure does not exist anymore. However, for Texas shown in Figure 8, when decreasing the depth from d = 4 (ACC=50%) to d = -0 . 35 (ACC=80%), the clusters of different classes/colors become wellseparated. Especially, the purple nodes spread randomly all over the embedding space when d = 4 but concentrate in one cluster when d = -0 . 35 . The spatial distribution of the node embeddings reveals that TEDGCN with positive and negative depths respectively models graph homophily and heterophily in a proper way.

(a)

d

=

-

2

, ACC=29.5%

(b)

d

= 0

, ACC=46.6%

(c)

d

= 5

, ACC=82.4%

(d) GCN, ACC=81.0%

(a)

d

=

-

0

.

35

, ACC=80.0%

(b)

d

= 2

, ACC=59.0%

(c)

d

= 4

, ACC=50.0%

(d) GCN, ACC=55.0%

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of graph heterophily in training Graph Neural Networks. Through graph spectral analysis, the authors decompose the graph Laplacian matrix into eigengraphs. Dwelling on the observation that low/high-frequency eigengraphs correspond to homophily/heterophily, the authors propose to use a continuous and trainable depth parameter to model the weight of the eigengraphs so the trained GNN can be adaptive to the level of heterophily in the target graph. The authors also realize the efficiency issue of eigendecomposition in large graphs and hence propose an approximation method that only considers the top-k largest/smallest eigenvectors and two trainable depth parameters to improve the efficiency. Experimental results show promising improvement over baseline methods on heterophily datasets.\n\nSTRENGTHS:\nI enjoy reading the paper as the authors explain the details and intuition behind the equations very well. The experiments are also solid verification of the proposed methods. The functional calculus trick to transfer the depth from the integer domain to the real domain is interesting and novel. The approximation method TEDGCN-D seems like a good alternative to using the full eigendecomposition and is organically developed from the original version.\n\nWEAKNESSES:\nMy primary concern lies in its comparison to existing work. The related work section only mentioned some work in the related field. However, a large body of work also realizes the importance of incorporating high-frequency passes in heterophilic graphs. How is the proposed method different from the existing ones? More specifically, note that in the formulation of the paper, the eigenvalues are shifted to (0,1] and d is uniform to both low/high frequency eigengraphs, meaning that d only shifts the focus on high/low frequency (amplifying the gap between eigenvalues too), what makes this different from other adaptive methods as the authors themselves point out in the paper?\n\nQUESTIONS:\n- line 187. I do not fully understand the negative depth explained for integer values. Since the proposed method only use either positive or negative depth, when does the model perform both positive message passing and negative message passing?\n- The value of K is max(1000, 0.1n), I think 0.1n is still a very large value. (1) what is the processing time for TEDGCN-D on larger dataset, e.g. ogbn-arxiv. (2) What happens when a smaller K is used for a larger graph? Is the top few components sufficient?\n- line 305. How is the optimal depth calculated? If it is a trainable parameter, how do you plot the full range of all depth?\n- Is using a single depth parameter d necessary? For example, you can use trainable d for all different eigengraphs. Note that this should bring minimal extra computation cost as the complexity is still the same. This potentially provide more flexible adaptation to different graphs.\n\nLIMITATIONS:\nThe proposed method uses spectrum and hence require additional computation for new and changed graphs. The cost of computing eigendecomposition for large graphs, even using the approximation method, is still unclear.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new GCN that is capable of handling both homophilic and heterophilic networks. This method can be interpreted as generalizing the depth of the GCN, which is generally a natural number and a hyperparameter, to an arbitrary real number that is a trainable parameter. The first version of this method is essentially a variant of the well-known SGC method that makes the exponent on the graph matrix (corresponding the the depth) a trainable parameter. This can be computationally expensive, so the paper also proposes a second version which extracts the lowest frequency components and highest frequency components of the graph, and gives two tunable depth parameters to both sets of components. Experiments show that these versions generally achieve comparable or superior performance on the standard set of benchmark networks for homophilic/heterophilic node classification, relative to some prior methods. Some other experiments examine the depth parameters found by the method on various networks, as well as other insights from the method.\n\nSTRENGTHS:\n- The paper is organized well and easy to read.\n- The proposed method is relatively simple as compared to similar recently proposed methods like GPR-GNN, but still demonstrates comparable or superior performance.\n- The paper proposes and evaluates both a conceptually simple version of the method, as well as a slightly more complex but scalable version.\n\nWEAKNESSES:\n- As the authors note, this method is only directly applicable to the most basic setting of static undirected graphs in the transductive setting. Many of the competing methods to which this paper provides comparisons also apply to the inductive setting.\n\n- While the simplicity of the method is appealing, since its dual variant (which is the only scalable variant) discards intermediate frequency components, it may struggle with graphs like Actor to which these components also seem crucial. The paper notes this.\n\n- While the number of datasets is quite reasonable, and these have been the most popular ones in this field recently, I would have liked to see evaluations on different datasets. 3 of the datasets are very small, and two others (Squirrel/Chameleon) have recently been shown to have other issues - see the paper cited below. That paper also introduces other heterophilic datasets, which could yield a better evaluation.\n\n- This is not really a weakness, but I do not see a significant conceptual contribution here prior to the introduction of the new method. There has been an abundance of recent papers on handling heterophily, and hence an abundance of discussion of various spectral filters and their interpretations.\n\n### Typos/minor\n- Line 101: \"and the corresponding eigengraph $\\mathbf{u}_1 \\mathbf{u}_1^\\top$ has an identical value $\\frac{1}{n}$ for all entries\" - isn't this only true for regular graphs?\n- Line 125: \"also corresponds a\"\n- Line 153-154: \"applying an arbitrary function on a graph Laplacian is equivalent to applying the same function only on the eigenvalue matrix\" - I think \"arbitrary\" has to be specified a bit here, e.g., this is true a matrix squaring function but not an entrywise squaring function\n- Table 2 caption: \"heterphilic\"\n\nQUESTIONS:\n- The Squirrel and Chameleon datasets were recently found to have a lot of erroneous duplicates (\"A critical look at the evaluation of GNNs under heterophily: are we really making progress?\" by Platonov et al.). Is this paper using the revised datasets or the originals?\n\n- The current method does not apply to the inductive setting, which is noted as a limitation of the work. Do the authors have any ideas on how the insights here could apply to that setting?\n\n- The depth parameter found by the single variant often seems to be interpretable. Are the two depth parameters found by the dual variant also interpretable?\n\nLIMITATIONS:\nLimitations are very briefly discussed in the appendix and pertain to the limited setting to which the method is applicable (static/transductive). There could be more discussion of limitations beyond this, such as limitations about the evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows a connection between the depth of a GCN and it's suitability for homophilic / heterophilic graphs, by analyzing graph spectra. It proposes to address heterophily with negative depth and presents a GNN architecture called TeDGCN which allows for trainable and negative depth. TeDGCN outperforms / is competitive with other GCN variants on a number of graph datasets. It further leads to a graph augmentation approach when the optimal depth is known.\n\nSTRENGTHS:\nThis paper tries to extend GCN with trainable real-valued depth. This has consequences for automatic tuning of the GCN depth, and for better addressing heterophilic datasets, both of which make it a significant novel contribution to the community. It establishes the connection between positive/negative depth with low/high frequency components of the graph signal and homophily/heterophily.\n\nThe analysis results in a new architecture TeDGCN. The paper further extends it to two variants, dealing with both low- and high-frequency components and addressing scalability challenges for large graphs. The method gives superior results, especially for heterophilic datasets. The trained depths are analyzed and also used to propose a graph augmentation mechanism which sometimes outperforms TeDGCN.\n\nThe paper is well-written and the ideas are presented clearly.\n\nWEAKNESSES:\nTo enable trainable depth, the paper operates on a simplification of GCN which removes the intermediate non-linearities. Thus the resulting model is a linear model, which limits the expressive power of the approach. Also, as a GCN variant it is not applicable to the inductive setting.\n\nIt is not clear whether the lessons from this paper can be applied to more advanced GNNs or hold more broadly. Essentially trainable depth results in an augmented graph. Can this augmented graph be used with non-linear GNNs to still capture heterophily?\n\nSome baselines with trainable depth (ODE-based GNNs) could be compared against. Is it possible to realize negative depth by inverting the graph Laplacian first?\n\nQUESTIONS:\nOn line 262 the split should be 60/20/20 right?\n\nRecent work [1] has shown pressing issues in old benchmarks of heterophilic datasets, which have been used here. Can you also include / replace with the results on the newer datasets proposed in [1]?\n\nThe same paper also shows that standard architectures like SAGE and GAT can outperform heterophily-specific architectures. Can you include these as baselines? Even if not adding datasets from [1], these baselines would be representative of models with higher expressive power than GCN or GCN-variants.\n\n[1] Platonov et. al. \"A critical look at the evaluation of GNNs under heterophily: Are we really making progress?\" ICLR 2023.\n\nLIMITATIONS:\nThe limitation to transductive problems is mentioned briefly in the appendix, but the limitation to linear models and expressive power is not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two algorithms to learn GNN with a trainable depth. At first, the authors exploit previous theoretical results about the correlation between frequency and zero crossings and the intuition that capturing heterophily needs more zero crossings to motivate adjusting the weights of eigen-graphs. Then it is natural to achieve this goal by tuning the depth of a GNN, where the authors propose to extend the possible space of depths to real numbers. To my knowledge, it is the first attempt to train a GNN with trainable and possibly negative depth, especially since the proposed algorithms are not built upon continuous-time diffusions. Extensive experiments are conducted, showing that the proposed algorithms are quite effective for heterophilic graphs, and they tend to learn the optimal depth.\n\nSTRENGTHS:\n1. This well-written paper enables me to pick up its core ideas effortlessly. Particularly, the provided examples are quite helpful for understanding the corresponding definitions and formulas.\n2. Learning a negative depth to encourage larger weights for eigen-graphs that provide more zero crossing interests me. More importantly, it is well-motivated and demonstrates significant advantages in capturing heterophilic graph patterns.\n3. The empirical studies seem to be comprehensive. At first, both homophilic and heterophilic graphs are included. I am glad to see that the authors honestly report the results on those four homophilic graphs, where the proposed algorithms have not achieved a consistent improvement. And it is great to see the dramatic improvements on heterophilic graphs, which strongly support the idea of this paper. Second, the detailed analysis confirms that the proposed algorithms can truly learn the optimal depth. This lets us attribute the successes in capturing heterophilic graph patterns to such a mechanism and its consequence of up-weighting zero crossings.\n4. I am quite fond of the interpretation of augmenting original graph structure, which perfectly obeys our intuition.\n\nWEAKNESSES:\n1. How to apply the proposed algorithms to large graphs is a critical question that has not been fully resolved.\n2. The motivation of proposed algorithms seems to be entirely built upon previous theoretical results. I am curious about what is exactly the contribution of this paper.\n3. To achieve a trainable and negative depth, the designed GNN is in a similar form to SGC, which also sacrifices some non-linearity (or expressiveness) to gain simplicity.\n4. The rescaling of the base into (0, 1) is reasonable and explainable, which is satisfactory to me. Yet, it would be better to justify its expressiveness and numerical properties as original bases (i.e., original frequencies).\n\nQUESTIONS:\nIn line 130, the cardinalities of examples are 1 and 2. As the considered graph is undirected, why not 2 and 4?\n\nLIMITATIONS:\nThere is a complexity analysis of the proposed algorithms, which can be regarded as a discussion of the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}