Title: LD2: Scalable Heterophilous Graph Neural Network with Decoupled Embeddings
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph neural networks

Scalability

Heterophilous Graphs

Non-Homophily
Abstract: Heterophilous Graph Neural Network (GNN) is a family of GNNs that specializes in learning graphs under heterophily, where connected nodes tend to have different labels. Most existing heterophilous models incorporate iterative non-local computations to capture node relationships. However, these approaches have limited application to large-scale graphs due to their high computational costs and challenges in adopting minibatch schemes. In this work, we study the scalability issues of heterophilous GNN and propose a scalable model, LD2, which simplifies the learning process by decoupling graph propagation and generating expressive embeddings prior to training. Theoretical analysis demonstrates that LD2 achieves optimal time complexity in training, as well as a memory footprint that remains independent of the graph scale. We conduct extensive experiments to showcase that our model is capable of lightweight minibatch training on large-scale heterophilous graphs, with up to $15\times$ speed improvement and efficient memory utilization, while maintaining comparable or better performance than the baselines.

LD 2 : Scalable Heterophilous Graph Neural Network with Decoupled Embeddings

Abstract

Heterophilous Graph Neural Network (GNN) is a family of GNNs that specializes in learning graphs under heterophily, where connected nodes tend to have different labels. Most existing heterophilous models incorporate iterative non-local computations to capture node relationships. However, these approaches have limited application to large-scale graphs due to their high computational costs and challenges in adopting minibatch schemes. In this work, we study the scalability issues of heterophilous GNN and propose a scalable model, LD 2 , which simplifies the learning process by decoupling graph propagation and generating expressive embeddings prior to training. Theoretical analysis demonstrates that LD 2 achieves optimal time complexity in training, as well as a memory footprint that remains independent of the graph scale. We conduct extensive experiments to showcase that our model is capable of lightweight minibatch training on large-scale heterophilous graphs, with up to 15 × speed improvement and efficient memory utilization, while maintaining comparable or better performance than the baselines. Our code is available at: https://github.com/gdmnl/LD2 .

1 Introduction

Graph Neural Networks (GNNs) combine graph management techniques and neural networks to learn from graph-structured data, and have shown remarkable performance in diverse graph processing tasks, including node classification [1, 2], link prediction [3, 4], and community detection [5, 6]. Common GNN models rely on the principle of homophily , which assumes that connected nodes tend to be similar to each other in terms of classes [7]. This inductive bias introduces additional information from the graph structure and improves model performance in appropriate tasks [8].

However, this assumption does not always hold in practice. A broad range of real-world graphs are heterophilous , where class labels of neighboring nodes usually differ from the ego node [9]. In such cases, the aggregation mechanism employed by conventional GNNs, which only passes messages from a node to its neighbors, may mix the information from non-homophilous nodes and cause them to be less discriminative. Consequently, the locality-based design is considered less advantageous or even potentially harmful in these applications [10, 11]. Various solutions have been proposed to address the heterophily problem, giving rise to a class of specialized GNNs known as heterophilous GNNs. Common strategies to address heterophily include discovering multi-hop or global graph relations [12, 13, 14, 15, 16, 17], and retrieving expressive node information through enhanced network architectures [18, 19, 20, 21, 22, 23].

Existing heterophilous GNNs are not scalable enough. Scalability has become a prominent concern in GNN studies. The ever-increasing sizes of graph data nowadays can easily exceed the memory limit of devices such as GPUs, rendering these models impractical for large-scale tasks [24, 25]. We observe that this issue is particularly critical in the context of heterophilous GNNs, due to an inherent conflict that most current models have not taken into account: heterophily-oriented designs usually rely on non-local information calculated by certain types of whole-graph operations. As the graph structure is involved, the time and memory overhead escalates substantially with the graph size. A recent investigation [26] reveals that all the evaluated full-graph GNNs run out of 24 GB GPU memory when applied to the million-scale graph wiki ( 1 . 77M nodes, 243M edges), while minibatch strategies often result in performance loss. It is thus crucial to develop GNNs scalable to large graphs while retaining the capability for heterophily.

LD 2 : a scalable solution via decoupling. In this work, we examine the scalability problem and propose LD 2 , a scalable GNN model for heterophilous graphs with Low-Dimensional embeddings and Long-Distance aggregation. The model highlights simplicity by decoupling graph dependency from iterative computations and solely learning from a set of precomputed embeddings. Derived from node attributes and graph topology, these novel embeddings are capable of aggregating node relations of varying objectives and distances in the graph into low-dimensional features. To facilitate the decoupled scheme, we specifically propose an algorithm to efficiently estimate all embeddings before training, which enjoys time complexity only linear to the graph scale and a guaranteed precision bound. After the precomputation, a simple but powerful multi-channel neural network is subsequently employed to learn from the extracted node features. Theoretical and empirical results showcase that the combination of embeddings effectively retrieves representations among heterophilous nodes. On the efficiency aspect, LD 2 is advantageous in its scalable design, including the straightforward minibatch scheme, optimal training and inference times, and the superior memory utilization.

Our contribution. (1) We propose LD 2 as a scalable GNN under heterophily, which removes the reliance on iterative train-time full-graph computations. The model realizes theoretically optimized training, highlighting time complexity that is only linear to the number of nodes O ( n ) and memory overhead independent of the graph scale. To the best of our knowledge, LD 2 is the first model achieving such optimization in the context of heterophilous GNNs. (2) We design an array of feature and topology embeddings by applying multi-hop discriminative propagation, encoding expressive node representation within a compact size. An end-to-end precomputation algorithm is proposed for efficient embedding calculation with a linear complexity. (3) We conduct comprehensive experiments to evaluate the efficacy and efficiency of LD 2 . On large-scale datasets, our model demonstrates 3 -15 × faster minibatch training and inference, and up to 5 × smaller memory footprint than state of the art, with comparable or better accuracy. Particularly, it completes training on the wiki dataset within 1 minute and inference in 0 . 1 seconds, demanding only 5GB of GPU memory.

2 Preliminary and Related Work

Graph Notation and Heterophily. In an undirected graph G = ( V, E ) with node set V and edge set E , the number of nodes, number of edges, and average degree are denoted by n = | V | , m = | E | , and d = m/n , respectively. The neighborhood of an ego node u ∈ V is the set N ( u ) = { v | ( u, v ) ∈ E } , and its degree d ( u ) = |N ( u ) | . The diagonal degree matrix is D = diag( d ( u 1 ) , d ( u 2 ) , · · · , d ( u n )) .

The graph connectivity is represented by the adjacency matrix A ∈ R n × n . We adopt the general graph normalization scheme [31, 32] with coefficients a, b ∈ [0 , 1] and ¯ A = D -a AD -b . The normalized adjacency matrix with self-loop edges is also frequently used, which is denoted as ˜ A = ( I + D ) -a ( I + A )( I + D ) -b , and the corresponding graph Laplacian matrix is ˜ L = I -˜ A . Each node u ∈ V is represented by an F -dimensional attribute vector X ( u ) , which composes the attribute matrix X ∈ R n × F . The graph heterophily is measured by the node homophily score [14], which is the average proportion of the neighbors with the same class of each node.

Vanilla and Sampling-based GNNs. Wesummarize our analysis of the time and memory complexity of related GNN models in Table 1. In general, a GNN recurrently computes the node representation H ( l ) in its l -th layer. For the vanilla GCN [1], the model input is the attribute matrix H (0) = X , and the layer representation is updated by H ( l +1) = σ ( ˜ AH ( l ) W ( l ) ) , l = 0 , 1 , · · · , L -1 , where W ( l ) is the trainable weight matrix each layer, σ ( · ) is the activation function, and L is the number of layers. For simplicity we assume the feature size F to be constant in all layers. Previous research [33, 31] points out that the operation of dominating expense in both GCN training and inference is the graph propagation ˜ A · H ( l ) , which can be regarded as repetitive sparse-dense matrix multiplications, resulting in a total complexity O ( LmF ) . The overhead for feature transformation by applying W ( l ) is O ( LnF 2 ) . These two procedures are iteratively performed for I epochs throughout training. In terms of memory usage, GCN typically requires O ( LnF + LF 2 ) space to store layer-wise node representations and weight matrices, respectively. For large-scale cases where n /greatermuch F , the overhead of dense node representations O ( LnF ) becomes the primary term [24].

The above analysis indicates that the scalability bottleneck of GCN lies in the time complexity of graph propagation as well as the memory overhead of full-graph representation. There is a large scope of GNNs attempting to address the issue by sampling techniques, which simplify the propagation by replacing the entire graph with subgraphs in minibatches [2, 27, 33]. For example, the widely-used GSAINT [27] incorporates L P -hop random walk sampling, reducing the in-memory representation to O ( L P Ln b F ) , where n b is the batch size. However, it is not applicable to the inference stage.

Full-graph Heterophilous GNNs. In the context of GNNs under heterophily, a large number of models augment the spatial graph convolution to a full-graph scheme, relying upon the complete graph topology to compute inter-node relationships. For instance, H2GCN [13] and MixHop [12] incorporate 2-hop propagation ˜ A 2 H ( l ) , while GeomGCN [14] and GloGNN [22] exploit hierarchical computation on non-local connections. These high-order calculations are shown to be effective in retrieving information beyond immediate neighbors, but come at the price of more complex propagation operations. Another common practice for non-homophilous design is altering transformation to learn from multiple features, i.e. channels. MixHop [12] mixes its multi-hop representation as H ( l +1) = σ ( H ( l ) W ( l ) 0 ‖ ˜ AH ( l ) W ( l ) 1 ‖ ˜ A 2 H ( l ) W ( l ) 2 ) , where ( · ‖ · ) denotes matrix concatenation. GGCN [19] and ACM [20] select frequency-based filters expressed by ˜ AHW l , ( I -˜ A ) HW h , IHW i , while GCNJK [21] records individual layer representations as channels. Denote the number of channels as C and L C = L + C , employing multi-channel learning increases the memory budget for node representations and weight matrices by a factor related to C or L C .

Alternatively, a line of approaches choose to apply learnable spectral filters for flexibility on nonhomophilous data, deriving specific propagation in the spectral domain for various graphs. GPRGNN [17] generalizes the multi-hop propagation as T = ∑ L P l =0 θ l ˜ A l , which inevitably involves the entire adjacency matrix for acquiring weight factors θ l . BernNet [34] and ChebNetII [35] respectively utilize orderL P Bernstein and Chebyshev polynomials with respect to ˜ L to approximate their spectral filters. These models focus relatively more on the expressiveness of graph propagation rather than the overall GNN architectural design, usually demanding additional cost for determining spectral filters and updating the learnable basis.

In spite of their advantageous capabilities, above full-graph designs hardly address the scalability bottleneck in GNN propagation, which can be observed from the O ( n ) or O ( m ) terms in their complexity. Recent studies also discover that heterophilous GNNs are naturally unsuitable for enforcing scalable training via sampling-based minibatching, since their distant or full-graph information is heavily overlooked in batches built on locality [11]. Evaluations show that simply fitting these models to learn from induced subgraph samples causes performance degradation [26]. Therefore, we believe these full-graph models targeting effectiveness are orthogonal to our study on scalability.

Decoupled GNNs. As the scalability of GNN is closely tied to the graph propagation, a promising approach to simplify the process is decoupling it from iterative training and efficiently computing it in advance. The representative two-stage model SGC [30] encodes graph information with X into an embedding matrix P = ˜ A L P · X , which is then input to a Multi-Layer Perceptron H ( L ) = MLP( P ) . [36, 32, 25] further generalize the embedding aggregation. Such decoupled GNN is considered optimal for training efficiency, as its time complexity O ( ILnF ) is identical to the simple MLP [31].

However, applying the decoupling technique to heterophilous GNNs is non-trivial due to the fullgraph relationships. To our knowledge, LINKX [26] is the only model conceptually similar to this scheme, removing graph-related propagation during training iterations and enabling solely node-wise minibatching. It exploits a simple architecture H ( L ) = MLP( XW X ‖ AW A ) , where the matrix A is used as an input feature in learning. The major drawback of this design is that it suffers from the O ( nF ) term in model size and O ( mF ) term in forward time, hindering its scalability to large graphs.

3 Method

In this section, we first present an overview of our LD 2 model, then respectively elaborate on the selection of the adjacency and feature embeddings. Lastly, an end-to-end scalable algorithm, namely A 2 Prop, is proposed to efficiently and concurrently compute all the embeddings.

3.1 LD 2 : A Decoupled Heterophilous GNN

In order to achieve superior time and memory scalability for heterophilous GNNs, we employ the concept of decoupling, which removes the dependency of graph adjacency propagation in training iterations. The main idea of our model is first generating embeddings from raw features including node attributes and adjacency in a precomputation stage. Then, these embeddings are taken as inputs to learn representations by a simple neural network model. We embrace the multi-channel architecture [37, 20] to enhance flexibility, where the input data is a list consisting of embedding matrices [ P 1 , P 2 , · · · , P C ] . Each embedding is separately processed and then merged in the network.

LD 2 utilizes diverse embeddings based on pure graph adjacency and node attributes, denoted as P A ( A ) and P X ( X , A ) , respectively. Both types of embeddings can be produced by our precomputation A 2 Prop following Algorithm 1. The initial layer of the LD 2 network applies a separate linear transformation to each embedding input, and the results are concatenated to form the representation matrix. Lastly, an L -layer MLP is leveraged for the classification task. The high-level framework of LD 2 is depicted in Figure 1 and can be expressed as follow:

Precompute : P A , P X = A 2 Prop( A , X ); Transform : H ( L ) = MLP( P A W A ‖ P X W X ) . (1)

Training/Inference Complexity. Our decoupled model design enables a simple on-demand minibatch scheme in training and inference, that only n b rows corresponding to the batch nodes in the embedding matrices are loaded into GPU and processed by the network transformation. For LD 2 with C channels, the GPU memory footprint is therefore bounded by O ( L C n b F + L C F 2 ) . It is worth noting that such complexity does not depend on the graph scale n or m . Consequently, the training is freely configurable with an arbitrary GPU memory budget. Regarding computation operations, the time complexity of forward inference through the graph is O ( LnF 2 ) , being just linear to n . As

n

F

X

A

∑

A

l

P

A

P

X

∑

A

l

H

(1)

H

(

L

)

||

Approximate Propagation

Feature Embedding

Feature Transformation

σ

W

X

W

A

Precomputation

Training/Inference

A

2

Prop

MLP

u

0

u

1

u

2

u

3

u

4

+

+

-

+

-

+

2-hop

2-hop

1-hop

1-hop

1-hop

1-hop

y

0

=0

y

1

=1

y

2

=0

y

3

=1

y

4

=0

Inverse 1-hop Laplacian Propagation

Constant 2-hop Adjacency Propagation

P

L2

(0)

P

L2

(2)

P

L2

(4)

P

H

(0)

P

H

(1)

P

H

(2)

P

H

(3)

P

H

(4)

<latexi sh

1_b

64="

AfG

K

MYjyzdCP

w5VrDLnU

>

BH

c

7T

N

EF

v

W

kq

m

o

RS

Z

u

/

I

9

+g

p

0

38

2

X

JO

Q

<latexi sh

1_b

64="

AfG

K

MYjyzdCP

w5VrDLnU

>

BH

c

7T

N

EF

v

W

kq

m

o

RS

Z

u

/

I

9

+g

p

0

38

2

X

JO

Q

the model complexity only contain essential operations of MLP-like transformation on nodes in the graph with no additional expense, this is the optimal scale with respect to the iterative training of GNN architectures.

3.2 Low-Dimensional Adjacency Embedding

Several studies reveal that, despite the feature information of nodes, the pure graph structure is equally or even more important in the context of heterophilous GNNs [9, 14, 26]. Particularly, the most informative aspects are often associated with 2-hop neighbors, i.e., 'neighbors of neighbors' of ego nodes. [13] proves that even under heterophily, the 2-hop neighborhood is expected to be homophilydominant. [38] also verifies that the 2-hop similarity is strongly relevant to GNN performance. We thence intend to explicitly model such topological information.

The 2-hop relation can be described by the 2-hop adjacency matrix A 2 . Note that as the sparse matrix A has m entries, the number of entries in A 2 is at the scale of O ( md ) , which indicates that directly applying 2-hop graph propagation in the training stage will demand even more expensive time and memory overhead to be scaled up. We instead propose an approximate scheme that seeks to prevent the 2-hop adjacency from explicit processing, and retrieves a low-dimensional but expressive embedding prior to training in the precomputation stage. In other words, we utilize the embedding to resemble 2-hop information which can be directly learned by the neural network transformation. Denote the F -dimensional embedding matrix as P A ∈ R n × F . We aim to minimize its approximation error in Frobenius norm ( ‖ · ‖ F ) :

The solution to Eq. (2) can be derived from the eigendecomposition of the symmetric matrix A 2 , that P ∗ A = U | Λ | 1 / 2 , where Λ = diag( λ 1 , · · · , λ F ) is the diagonal matrix with topF eigenvalues λ 1 ≥ λ 2 ≥ · · · ≥ λ F , and U ∈ R n × F is the matrix consisting of corresponding orthogonal eigenvectors. The eigenvalues are also called frequencies of the graph, and large eigenvalues of the adjacency matrix refer to low-frequency signals in the graph spectrum.

Spectral Analysis. Let A 2 ( u, v ) be the entry ( u, v ) of matrix A 2 . Its diagonal degree matrix is D 2 = diag( d 2 ( u 1 ) , d 2 ( u 2 ) , · · · , d 2 ( u n )) , where d 2 ( u ) = ∑ v ∈ V A 2 ( u, v ) . Denote P A ( u ) as the F -dimensional adjacency embedding vector of node u . We show that the embedding matrix P ∗ A defined by Eq. (2) is also the solution to the following optimization problem:

This is because ∑ u,v A 2 ( u, v ) ‖ P ( u ) -P ( v ) ‖ 2 = 2 ∑ u d 2 ( u ) ‖ P ( u ) ‖ 2 -2 ∑ u,v A 2 ( u, v ) P ( u ) P ( v ) = 2 tr( P /latticetop D 2 P -P /latticetop A 2 P ) . As P /latticetop D 2 P is fixed, finding the minimum of Eq. (3) is equivalent to optimizing max P P /latticetop A 2 P , of which the solution is exactly P ∗ A according to the property of eigenvectors. Equation (3) implies that, 2-hop neighbors ( u, v ) , t ∈ N ( u ) , v ∈ N ( t ) in the graph will share similar embeddings P A ( u ) and P A ( v ) .

In fact, the low-dimensional embedding P ∗ A can be interpreted as the adjacency spectral embedding of the 2-hop graph A 2 . Graph spectral embedding is a technique concerning the low-frequency spectrum of a graph, and is employed in tasks such as graph clustering [39]. As P A corresponds to the dominant eigenvalues of A 2 , the embedding provides an approximate representation of the 2-hop neighborhoods based on the overall graph topology. Alternatively, if we regard the adjacency information solely as features input into the network like LINKX, P A correlates to the uncentered principal components of matrix A . Therefore, learning a linear transformation P A W A with weight matrix W A ∈ R F × F in LD 2 is the same expressive as the rankF approximation of AW A 0 in LINKX, where W A 0 ∈ R n × F , but with a less computational cost independent to the graph scale. Compared to other works attempting to generate graph embeddings based on graph geometric or similarity measures [40, 41, 14, 16, 19, 42], our approach offers the advantages of lower dimensionality and efficient calculation as demonstrated in Section 3.4.

3.3 Long-Distance Feature Embedding

Decoupling the node features through approximate propagation has been extensively studied in regular GNNs with various schemes [28, 30, 36, 29, 43, 32, 44]. Nonetheless, these approaches are

based on the homophily assumption and focus on local neighborhoods. In order to apply decoupled propagation to heterophilous graphs and exploit the multi-channel ability of our model, we formulate the general form of approximate propagation as the weighted sum of powers of a propagation matrix applied to the input feature, i.e., P X = ∑ L P l =1 θ l T l X . Examples of propagation matrix T include ˜ A and ˜ L , which respectively correspond to aggregative and discriminative operations.

LD 2 utilizes the following channels jointly as input embeddings: (1) inverse summation of 1-hop improved Laplacian propagations P X,H = 1 L P,H ∑ L P,H l =1 ( I + ˜ L ) l X , ( θ l = 1 , T = I + ˜ L ) ; (2) constant summation of 2-hop adjacency propagations P X,L 2 = 1 L P,L 2 ∑ L P,L 2 l =1 ¯ A 2 l X , ( θ l = 1 , T = ¯ A 2 ) ; (3) raw node attributes P X, 0 = X .

Intuitively, the first two channels perform distinct topology-based propagations on node feature X , and employ inverse or constant summation to aggregate multi-hop information, in contrast to the local decaying summation ( θ l → 0 when l →∞ ) commonly adopted in homophilous GNNs. Hence, such summations are suitable for retrieving long-range information under heterophily. The raw matrix X is also directly used as one input channel to depict node identity, which is a ubiquitous practice known as the skip connection or all-pass filter in heterophilous GNNs [12, 45, 20, 22].

Illustrated in Figure 2, the inverse embedding P X,H is based on the intuition that, as neighbors tend to be different from the ego node, their features are also dissimilar. Hence in propagation, the embedding of the ego node should contain the previous embedding of itself, as well as the inverse of adjacent embeddings, which is exactly the interpretation of propagating node features by graph Laplacian matrix ˜ L = I -˜ A , while an additional identity matrix is applied to balance the embedding distribution. The second embedding P X,L 2 performs a 2-hop propagation through the graph and aggregates the results of multi-scale neighbors. It echoes the earlier statement on the importance of 2-hop adjacency from the feature aspect. Note that for P X,L 2 , the employed adjacency matrix is ¯ A which escapes self-loops, since it is shown to be relatively favorable for capturing non-local homophily in multi-hop propagations compared with ˜ A [13, 23].

Spectral Analysis. Assume that ‖ X ( u ) ‖ = ‖ P ( u ) ‖ = 1 . We first examine the following regularization problem optimizing the embedding P based on input X for homophilous graphs [43]:

Differentiating the objective function with respect to P leads to ( I -˜ A ) P -X = 0 . Therefore the closed-form solution is P ∗ X,L = ( I -˜ A ) -1 X = ∑ ∞ l =0 ˜ A l X . In practical implementation, a limited L P,L -hop summation is used instead due to the over-smoothing issue that the infinite form converges to identical node-wise embeddings. This Markov diffusion kernel P X,L = 1 L P,L ∑ L P,L l =0 ˜ A l X is investigated in S 2 GC [43] as an approach for balancing locality and multi-hop propagation, functioning as a low-pass filter to the signal X but also preserves high frequency. Its interpretation can be observed from Eq. (4), that it simultaneously minimizes the embedding difference of neighboring nodes as well as the approximation closeness to the input feature X .

To obtain the channel P X,L 2 used in LD 2 , we introduce the low-frequency regularization preferably to 2-hop adjacency in the graph, as 1-hop neighbors exhibit heterophily. Therefore, replacing ˜ A ( u, v ) in Eq. (4) with ¯ A 2 ( u, v ) yields our constant 2-hop embedding P X,L 2 . It shares similar spectral properties with S 2 GC for acting as a low-pass filter in 2-hop neighborhoods, while maintaining certain longdistance knowledge thanks to the multi-scale aggregation. The other channel in feature embedding, i.e. the Laplacian propagation, can be derived as P X,H = ( I + ˜ L )( I -˜ A ) -1 X = ( I + ˜ L ) P X,L . Based on the above analysis, the embedding P X,L contains multi-hop neighborhood information, while ( I + ˜ L ) can be seen as the improved Laplacian operator extracting the high-frequency components. The embedding P X,H thus serves as a high-pass filter focusing on discriminative structures in a non-local manner. In terms of spatial domain interpretation, such high-frequency information corresponds to the fine-grained embedding differences between two nodes [18, 46]. It is noticeable that these three channels P X,L 2 , P X,H , P X, 0 respectively represent low-pass, high-pass, and all-pass propagations through the graph while addressing heterophily. Combining them as inputs to the neural network benefits the model performance with expressive information at various distances including identity, local, and global perspectives.

3.4 Approximate Adjacency Propagation Precomputation

Conventionally, calculating the graph propagation ˜ A · P for an arbitrary feature matrix P is conducted by the sparse-dense matrix multiplication. However, such an approach does not recognize the property of the adjacency matrix ˜ A , that it can be represented by the adjacency list of nodes, and non-zero values in its data are solely determined by node degrees. Furthermore, since the propagation result is subsequently processed by the neural network, it is not necessary to be precise as the model is robust to handle noisy data [46, 47]. We first define the precision bound for approximate embedding:

Definition 3.1 ( Approximate Vector Embedding ) . Given a relative error bound 0 < /epsilon1 < 1 , a norm threshold δ > 0 , and a failure probability 0 < φ < 1 , the estimation ˆ P ( u ) for an arbitrary embedding vector P ( u ) should satisfy that, for each u ∈ V with ‖ P ( u ) ‖ > δ , ‖ P ( u ) -ˆ P ( u ) ‖ ≤ /epsilon1 · ‖ P ( u ) ‖ with probability at least 1 -φ .

Graph power iteration algorithm is the variant of power iteration particularly applied for calculating powers of adjacency matrix A [48]. In essence, the algorithm can be derived by maintaining a residue R ( l ) ( u ) that holds the current l -hop propagation results for each node, and iteratively updating the next-hop residues of neighboring nodes R ( l +1) ( v ) , v ∈ N ( u ) for all nodes u . For each iteration, the reserve ˆ P ( l ) is also added up and converges to an underestimation of P .

We propose Algorithm 1 for our specific scenario, namely Approximate Adjacency Propagation (A 2 Prop). Based on power iteration, our algorithm is greatly generalized to accommodate normalized adjacency, feature vectors for nodes, and a limited number of hops. We show that the algorithmic output can be bounded by Definition 3.1. For L P iterations, denote the acceptable error per entry for push as δ P , the matrix-wise absolute error is ‖ P -ˆ P ‖ 1 , 1 ≤ ∑ L P f =1 ∑ F f =1 ∑ u ∈ V d ( u ) δ P = L P mF δ P . By setting δ P = /epsilon1δ /L P m , the estimation ˆ P satisfies Definition 3.1.

Approximate Feature Embedding. The feature embedding formed as P X = ∑ L P l =0 θ l T l X can be computed by iteratively applying graph power iterations to the initial residue R (0) = X . The implicit propagation behavior is described by matrix T . For example, for Laplacian propagation T = I + ˜ L to node u , the embeddings from the previous iteration are aggregated as R ( l +1) ( u ) = 2 R ( l ) ( u ) -∑ v ∈ N ( u ) R ( l ) ( v ) /d a ( u ) d b ( v ) = ∑ v ∈ N ( u ) ∪ { u } α L ( u,v ) d a ( u ) d b ( v ) · R ( l ) ( v ) . Here α T ( u, v ) is a propagation factor for unifying the aggregation by T , that α L ( u, u ) = 2 d a + b ( u ) , α L ( u, v ) = -1 , v ∈ N ( u ) . For propagation ˜ A and ¯ A , the factor is α A ( u, v ) = 1 and α A ( u, u ) = 1 , 0 , respectively.

In each iteration l , the reserve is updated after propagation according to the coefficient θ l to sum up corresponding embeddings. Intuitively, one multiplication of ¯ A 2 is equivalent to two iterations of ¯ A propagation. Hence for P X,L 2 there is θ l = l mod 2 = 0 , 1 , 0 , 1 , · · · under the summation scheme

2

2

4

6

8

10

12

14

16

18

20

80

81

82

83

84

85

86

Test Acc (%)

PX,

0

PA

PX,L

2

PX,H

PX,L

2

||

PX,H

PX

PA

||

PX

(a) genius

2

4

6

8

10

12

14

16

18

20

60

65

70

75

Test Acc (%)

PX,

0

PA

PX,L

2

PX,H

PX,L

2

||

PX,H

PX

PA

||

PX

(b) pokec

in Algorithm 1. Since all embeddings we consider are constant, that is, θ l ∈ { 0 , 1 } , the reserve can be simply increased without the rescaling terms in more general cases such as [32].

Approximate Adjacency Embedding. The adjacency embedding is represented by leading eigenvectors P A = U | Λ | 1 / 2 . This eigendecomposition of A 2 can be solved by the truncated power iteration [49]: Initialize the n × F residue by i.i.d. Gaussian noise R (0) = N (0 , 1) . For each iteration l , firstly multiply the residue by A 2 as R ( l +1) = A 2 R ( l ) ; then, perform column-wise normalization to the residue orthonormalize ( R ( l +1) ) so that its columns are orthogonal to each other and of L2 norm 1 . After convergence, the matrix satisfies A 2 R ( L P ) = R ( L P ) Λ within the error bound, which leads to the estimated output ˆ U = R ( L P ) , ˆ P A = ˆ U | ˆ Λ | 1 / 2 .

Similarly, the 2-hop power iteration of P A can be merged with those for P X with a shared maximal iteration L P , and orthonormalization is conducted every two A iterations. When the algorithm converges with error bound δ , the number of iteration follows L P = O (log( F/ δ ) / (1 -| λ F +1 / λ F | )) . By selecting proper values for F and δ , the algorithm produces satisfying results within L P iterations.

Precomputation Complexity. Since A 2 Prop serves as a general approximation for various adjacencybased propagations, the computation of all feature channels can be performed simultaneously in a single run. The memory overhead of the algorithm is mainly the residue and reserve matrices for C embedding channels, which is O ( CnF ) in total. Note that A 2 Prop precomputation is performed in the main memory, and benefits from a less-constrained budget compared to GPU memory.

For each A 2 Prop iteration, neighboring connections are accessed for at most m times. The time complexity of Algorithm 1 can thus be bounded by O ( L P mF ) . In addition, its loops over nodes and features can be parallelized and vectorized to reduce execution time. Moreover, the graph power iteration design is also amendable for further enhancements, such as reduction to sub-linear complexity [50, 25], better memory utilization [51, 52], and precision-efficiency trade-offs [53, 54]. We leave these potential improvements on A 2 Prop for future work.

4 Experimental Evaluation

We implement the LD 2 model and evaluate its performance from the perspectives of both efficacy and scalability. In this section we highlight key empirical results compared to minibatch GNNs on large-scale heterophilous graphs, while parameter settings, further experiments, and subsequent discussions can be found in the Appendix.

4.1 Experiment Setting

Datasets. We mainly perform experiments on million-scale and above heterophilous datasets [26, 55] for the transductive node classification task, with the largest available graph wiki ( m = 243M ) included. Evaluations on more homophilous and heterophilous graphs can be found in Appendices D and E. We leverage settings as per [26] such as the random train/test splits and the induced subgraph testing for GSAINT-sampling models.

Baselines. We focus on GNN models applicable to minibatch training in our evaluation regarding scalability, and hence most full-batch networks mentioned in Section 2 are excluded in the main experiments, while more comprehensive results for both minibatch and full-batch models are in Appendices D and E. Conventional baselines in the main experiments include MLP which only processes node attributes without considering graph topology, as well as PPRGo [29] and SGC [30] representing decoupled schemes for traditional graph propagation. For GNNs under non-homophily, we investigate GCNJK-GS [21] and MixHop-GS [12], where GSAINT random walk sampling [27] is utilized to empower the original backbone models for minibatching. LINKX is the decoupled heterophilous GNN proposed by [26]. Simple i.i.d. node batching is adopted for decoupled networks. Explorations on the model settings are displayed in Appendix C.

Evaluation Metrics. We uniformly use classification accuracy on the test set to measure network effectiveness. Note that since the datasets are updated and the minibatch scheme is employed, results may be different from their original works. In order to evaluate scalability performance, we conduct repeated experiments and record the network training/inference time and peak memory footprint as efficiency metrics. For precomputed methods, we consider the learning process combining both

∗ Inference time of GSAINT sampling is less precise since they are conducted on induced subgraphs smaller than the raw graph.

precomputation and training. Evaluations are conducted on a machine with 192GB RAM, two 28-core Intel Xeon CPUs (2.2GHz), and an NVIDIA RTX A5000 GPU (24GB memory).

4.2 Performance Comparison

The main evaluations of LD 2 and baselines on 8 large heterophilous datasets are presented in Tables 2 and 3 for effectiveness and efficiency metrics, respectively. As an overview, our model demonstrates its scalability in completing training and inference with fast running speed and efficient memory utilization, especially on large graphs. At the same time, it achieves comparable or superior prediction accuracy against the state-of-the-art minibatch heterophilous GNNs on most datasets.

Time Efficiency. We first highlight the scalability performance of our LD 2 model. Specifically, compared to heterophilous benchmarks on the four largest graphs with million-scale data, LD 2 speeds up the minibatch training process by 3-15 times, with an acceptable precomputation cost. Its inference time is also consistently below 0.1 seconds. The outstanding efficiency of LD 2 is mainly attributed to the simple model architecture that removes graph-scale operations while ensuring rapid convergence. In contrast, the execution speeds of MixHop and LINKX are highly susceptible to node and edge sizes, given their design dependency on the entire input graph. The extensive parameter space also causes them to converge slower, necessitating relatively longer training times. PPRGo shows limited scalability due to the costly post-transformation propagation. The superiority of LD 2 efficiency even holds when compared to simple methods such as MLP and SGC, indicating that the model is favorable for incorporating extra heterophilous information with no significant additional overhead. The empirical results affirm that LD 2 exhibits optimized training and inference complexity at the same level as simple models.

Memory Footprint. LD 2 remarkably reduces run-time GPU memory consumption. As the primary overhead only comprises the model parameters and batch representations, it enables flexible configuration of the model size and batch size to facilitate powerful training. Even for the largest graph

wiki with n = 1 . 77M and F = 600 , the footprint remains below 5GB under our hyperparameter settings. Other heterophilous GNNs, though adopting the minibatch scheme, experience high memory requirements and even occasionally encounter out-of-memory errors during experiments, as their space-intensive graph propagations are executed on the GPU. Consequently, when the graph scales up, they can only be applied with highly constrained model capacities to conserve space, potentially resulting in compromised performance.

Test Accuracy. With regard to efficacy, LD 2 achieves top testing accuracy on 6 out of 8 heterophilous graphs and comparable performance on the remaining ones. It also consistently outperforms the sampling-based GCNJK and MixHop, as well as conventional GNNs. Particularly, by extracting embeddings from not only node features but pure graph topology as well, LD 2 obtains significant improvements over feature-based networks on datasets such as genius, snap-patents, and wiki, demonstrating the importance of pure graph information in heterophilous learning. We deduce that the relatively suboptimal accuracy on penn94 may be correlated with the difficulty of fitting one-hot encoding features into informative embeddings, as explored in Appendix G. Consistent with the previous studies [26], regular GNN baselines suffer from performance loss on most heterophilous graphs, while MLP achieves comparably high accuracy when node attributes are discriminative enough. For non-homophilous models GCNJK and MixHop, the minibatch scheme hinders them from reaching higher results because of the neglect of their full-graph relationships.

4.3 Effect of Parameters

To gain deeper insights into the multi-channel embeddings of LD 2 , in Figure 3 we explore the effect of embeddings channels and propagation hops which are critical to our model design, while more discussions on other parameters and factors are displayed in Appendix F.

Embedding Channels. Lines in Figure 3 represent the results of learning on separate inputs on two representative datasets genius and pokec. It can be observed that different graphs imply varying patterns when embedding channels and propagation hops are changed. For the genius dataset where raw node attributes already achieve an accuracy above 82%, applying the other two feature embeddings further improves the result. While the adjacency embedding alone shows secondary performance, integrating it with other channels proves beneficial. In comparison, on pokec, it is the inverse embedding P X,H that becomes the key contributor. The empirical evaluation supports our design that by adopting multi-channel and heterophily-oriented embeddings, LD 2 benefits from learning both topology and feature for a more comprehensive understanding of the graph data.

Propagation Hops. As elaborated in Section 3.4, propagation hops L P determines the number of iterations in Algorithm 1. Particularly, for the approximate adjacency embedding P A , it also affects the convergence of decomposition. As shown by the brown dashdotted lines in Figure 3, the accuracy typically becomes stable when L P > 8 , indicating the utility of the low-dimensional approximation in producing effective topology embedding within limited iterations. For the multi-channel scheme in general, as the graph scale increases, employing more propagation hops becomes advantageous in capturing distant information. Aligned with our analysis, above observation validates that LD 2 is powerful in capturing implicit information of various frequencies and scales that is important in the presence of heterophily.

5 Conclusion

In this work, we propose LD 2 , a scalable GNN design for heterophilous graphs, that leverages long-distance propagation to capture non-local relationships among nodes, and incorporates lowdimensional yet expressive embeddings for effective learning. The model decouples full-graph dependency from the iterative training, and adopts an efficient precomputation algorithm for approximating multi-channel embeddings. Theoretical and empirical evidence demonstrates its optimized training characteristics, including time efficiency with a complexity linear to O ( n ) , and GPU memory independence from the graph size n and m . As a noteworthy result, LD 2 successfully applies to million-scale datasets under heterophily, with learning times as short as 1 minute and GPU memory expense below 5 GB . We also recognize the current limitations of our work including potential accelerations for precomputation and adaptability to diverse feature patterns. Detailed limitations and broader impacts are addressed in the Appendix.

Acknowledgments and Disclosure of Funding

This research is supported by Singapore MOE AcRF Tier-2 funding (MOE-T2EP20122-0003), NTU startup grant (020948-00001), and the Joint NTU-WeBank Research Centre on FinTech. Xiang Li is supported by National Natural Science Foundation of China No. 62202172 and Shanghai Science and Technology Committee General Program No. 22ZR1419900. Jieming Shi is supported by Hong Kong RGC ECS No. 25201221 and National Natural Science Foundation of China No. 62202404. We also thank Yuyuan Song for contributing to the experiments in this paper.

References

Kipf, T. N., M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations . 2017.

Hamilton, W. L., R. Ying, J. Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 1025-1035. 2017.

Zhang, M., Y. Chen. Link prediction based on graph neural networks. Advances in Neural Information Processing Systems , 31:5165-5175, 2018.

Ying, R., R. He, K. Chen, et al. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 974-983. 2018.

Al-Rfou, R., B. Perozzi, D. Zelle. Ddgk: Learning graph representations for deep divergence graph kernels. In The World Wide Web Conference , pages 37-48. 2019.

Chen, Z., J. Bruna, L. Li. Supervised community detection with line graph neural networks. 7th International Conference on Learning Representations , 2019.

McPherson, M., L. Smith-Lovin, J. M. Cook. Birds of a feather: Homophily in social networks. Annual Review of Sociology , 27(1):415-444, 2001.

Battaglia, P. W., J. B. Hamrick, V. Bapst, et al. Relational inductive biases, deep learning, and graph networks. arXiv:1806.01261, 2018.

Altenburger, K. M., J. Ugander. Monophily in social networks introduces similarity among friends-offriends. Nature Human Behaviour , 2(4):284-290, 2018.

Breuer, A., R. Eilat, U. Weinsberg. Friend or faux: Graph-based early detection of fake accounts on social networks. In Proceedings of The Web Conference 2020 , WWW'20, page 1287-1297. ACM, 2020.

Zheng, X., Y. Liu, S. Pan, et al. Graph neural networks for graphs with heterophily: A survey. arXiv:2202.07082, 2022.

Abu-El-Haija, S., B. Perozzi, A. Kapoor, et al. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In 36th International Conference on Machine Learning , vol. 97. PMLR, 2019.

Zhu, J., M. Heimann, Y. Yan, et al. Beyond homophily in graph neural networks: Current limitations and effective designs. In 33rd Advances in Neural Information Processing Systems , page 12. 2020.

Pei, H., B. Wei, K. C.-C. Chang, et al. Geom-gcn: Geometric graph convolutional networks. International Conference on Learning Representations , 2020.

Zhu, J., R. A. Rossi, A. Rao, et al. Graph neural networks with heterophily. Proceedings of the AAAI Conference on Artificial Intelligence , 35(12):11168-11176, 2021.

Suresh, S., V. Budde, J. Neville, et al. Breaking the limit of graph neural networks by improving the assortativity of graphs with local mixing patterns. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , 2021.

Chien, E., J. Peng, P. Li, et al. Adaptive universal generalized pagerank graph neural network. In 9th International Conference on Learning Representations . 2021.

Bo, D., X. Wang, C. Shi, et al. Beyond low-frequency information in graph convolutional networks. Proceedings of the AAAI Conference on Artificial Intelligence , 35(5):3950-3957, 2021.

Yan, Y., M. Hashemi, K. Swersky, et al. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. In 22nd IEEE International Conference on Data Mining . 2022.

Luan, S., C. Hua, Q. Lu, et al. Revisiting heterophily for graph neural networks. In 36th Advances in Neural Information Processing Systems . 2022.

Xu, K., C. Li, Y. Tian, et al. Representation learning on graphs with jumping knowledge networks. In 35th International Conference on Machine Learning , vol. 80. PMLR, 2018.

Li, X., R. Zhu, Y. Cheng, et al. Finding global homophily in graph neural networks when meeting heterophily. In 39th International Conference on Machine Learning . 2022.

Maurya, S. K., X. Liu, T. Murata. Simplifying approach to node classification in graph neural networks. Journal of Computational Science , 62:101695, 2022.

Peng, J., Z. Chen, Y . Shao, et al. Sancus: Staleness-aware communication-avoiding full-graph decentralized training in large-scale graph neural networks. Proceedings of the VLDB Endowment , 15(9):1937-1950, 2022.

Liao, N., D. Mo, S. Luo, et al. Scara: Scalable graph neural networks with feature-oriented optimization. Proceedings of the VLDB Endowment , 15(11):3240-3248, 2022.

Lim, D., F. Hohne, X. Li, et al. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. In 34th Advances in Neural Information Processing Systems . 2021.

Zeng, H., H. Zhou, A. Srivastava, et al. Graphsaint: Graph sampling based learning method. In International Conference on Learning Representations . 2019.

Klicpera, J., A. Bojchevski, S. Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. 7th International Conference on Learning Representations , pages 1-15, 2019.

Bojchevski, A., J. Klicpera, B. Perozzi, et al. Scaling graph neural networks with approximate pagerank. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2464-2473. ACM, 2020.

Wu, F., A. Souza, T. Zhang, et al. Simplifying graph convolutional networks. In K. Chaudhuri, R. Salakhutdinov, eds., Proceedings of the 36th International Conference on Machine Learning , vol. 97, pages 6861-6871. 2019.

Chen, M., Z. Wei, B. Ding, et al. Scalable graph neural networks via bidirectional propagation. 33rd Advances in Neural Information Processing Systems , 2020.

Wang, H., M. He, Z. Wei, et al. Approximate graph propagation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1686-1696. ACM, 2021.

Chiang, W.-L., X. Liu, S. Si, et al. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 257-266. 2019.

He, M., Z. Wei, Z. Huang, et al. Bernnet: Learning arbitrary graph spectral filters via bernstein approximation. In 34th Advances in Neural Information Processing Systems . 2021.

He, M., Z. Wei, J.-R. Wen. Convolutional neural networks on graphs with chebyshev approximation, revisited. In 35th Advances in Neural Information Processing Systems . 2022.

Gasteiger, J., S. Weißenberger, S. Günnemann. Diffusion improves graph learning. In 32nd Advances in Neural Information Processing Systems . 2019.

Wang, X., M. Zhu, D. Bo, et al. Am-gcn: Adaptive multi-channel graph convolutional networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD '20, page 1243-1253. Association for Computing Machinery, 2020.

Cavallo, A., C. Grohnfeldt, M. Russo, et al. 2-hop neighbor class similarity (2ncs): A graph structural metric indicative of graph neural network performance. In 3rd Workshop on Graphs and More Complex Structures for Learning and Reasoning (GCLR) at AAAI 2023 . 2022.

Belkin, M., P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation , 15(6):1373-1396, 2003.

Page, L., S. Brin, R. Motwani, et al. The pagerank citation ranking: Bringing order to the web. Tech. rep., 1999.

Liao, R., Z. Zhao, R. Urtasun, et al. Lanczosnet: Multi-scale deep graph convolutional networks. In International Conference on Learning Representations . 2019.

Liu, H., N. Liao, S. Luo. Simga: A simple and effective heterophilous graph neural network with efficient global aggregation. arXiv e-prints , 2023.

Zhu, H., P. Koniusz. Simple spectral graph convolution. In 9th International Conference on Learning Representations . 2021.

Zhu, Z., J. Peng, J. Li, et al. Spiking graph convolutional networks. In Proceedings of the 31st International Joint Conference on Artificial Intelligence , pages 2434-2440. 2022.

Ming, C., Z. Wei, Z. Huang, et al. Simple and deep graph convolutional networks. In 37th International Conference on Machine Learning , vol. 119, pages 1703-1713. PMLR, 2020.

Zhu, M., X. Wang, C. Shi, et al. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021 , pages 1215-1226. ACM, 2021.

Ma, Y., X. Liu, T. Zhao, et al. A unified view on graph neural networks as graph signal denoising. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 1202-1211. ACM, 2021.

Berkhin, P. A survey on pagerank computing. Internet Mathematics , 2(1):73-120, 2005.

Golub, G. H., C. F. V. Loan. Power iterations. In Matrix Computations , chap. 8.2, pages 450-457. JHU Press, 2012.

Wang, S., R. Yang, X. Xiao, et al. Fora: Simple and effective approximate single-source personalized pagerank. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , Part F1296:505-514, 2017.

Wang, K., S. Luo, D. Lin. River of no return: Graph percolation embeddings for efficient knowledge graph reasoning. arXiv e-prints , 2023.

Fang, P., A. Khan, S. Luo, et al. Distributed graph embedding with information-oriented random walks. Proceedings of the VLDB Endowmen , 16(7):1643-1656, 2023.

Wu, H., J. Gan, Z. Wei, et al. Unifying the global and local approaches: An efficient power iteration with forward push. In Proceedings of the 2021 International Conference on Management of Data , vol. 1, pages 1996-2008. 2021.

Mo, D., S. Luo. Agenda: Robust personalized pageranks in evolving graphs. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management . QLD, Australia, 2021.

Oleg Platonov, Denis Kuznedelev, Michael Diskin, et al. A critical look at evaluation of gnns under heterophily: Are we really making progress? In 11th International Conference on Learning Representations . 2023.

Wang, X., M. Zhang. How powerful are spectral graph neural networks. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, S. Sabato, eds., Proceedings of the 39th International Conference on Machine Learning , vol. 162, pages 23341-23362. PMLR, 2022.

Sen, P., G. Namata, M. Bilgic, et al. Collective classification in network data. AI Magazine , 29(3):93, 2008.

Hu, W., M. Fey, M. Zitnik, et al. Open Graph Benchmark: Datasets for Machine Learning on Graphs. 33rd Advances in Neural Information Processing Systems , 2020.

Ying, C., T. Cai, S. Luo, et al. Do transformers really perform bad for graph representation? In 34th Advances in Neural Information Processing Systems . 2021.

Grover, A., J. Leskovec. Node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , page 855-864. ACMZ, 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to optimize the design of graph neural networks on large-scale heterophilous graphs. It proposes a new framework called LD2, which decouples the node feature embedding and topology embedding. By obtaining a low-dimensional adjacency embedding and long-distance feature embedding through pre-computing, it achieves a faster and better performance for the mini-batch training.\n\nSTRENGTHS:\n1. The problem studied by this paper is interesting, The model efficiency is not well-studied on the large-scale heterophilous graphs and this research is meaningful to the community.\n2. The paper is well-organized and clearly presented. The authors make a thorough time complexity analysis and give sound solutions. Each component of the model is explained with reasonable logic.\n3. The experimental results look promising, with a clear advantage in the test accuracy and computational cost.\n\nWEAKNESSES:\n1. The comparison between LD2 and other models is not fair. The core component of LD2 is its precomputation stage. However, from its code, this part is implemented in C++ while other GNNs are implemented in Python.  The computation of $A^2$ and any operation involving it could be very expensive. Usually, $A^2$ could be $10$x or $100$x denser than $A$, and that is why most GNNs do not use it on large-scale graphs. Therefore, first, the time/space complexity for the precomputation of LD2 in Table 1 cannot reflect the actual computational cost (ignoring the difference between $A$ and $A^2$), and second, the time comparison between LD2 and other models is unfair due to the different programming languages.\n\n2. Although LD2 is claimed to be a scalable heterophilous GNN, its main contribution (precomputation) is closer to node embedding methods. LD2 combines and simplifies some existing techniques, which makes its contribution limited. I think the authors should also compare the embedding obtained from precomputation with some node embedding methods like node2vec. I expect there will be some tradeoff, but the discussion for this part is totally missing.\n\n3. The authors miss the baseline [1].\n\n[1] Sunil Kumar Maurya, Xin Liu, and Tsuyoshi Murata. Simplifying approach to node classification in graph neural networks. Journal of Computational Science, pp. 101695, 2022.\n\nQUESTIONS:\n1. I do not quite understand the formula in line 270: $R^{(l+1)}(u) = R^{(l+1)}(u) + R^{(l)}(u)$, what does it mean?\n\n2. It looks like Line 4 of algorithm 1 contains a typo (shall be the union rather than an intersection).\n\n3. Why are the dataset statistics of snap-patents and wiki different from that in [1]?\n\n4. Why is the batch size of LD2 larger than LINKX in most datasets? And why is the performance of full-batch LINKX (Table 7) different from that in [1]? I am not sure if the authors reproduce the correct results of LINKX.\n\n[1] Lim, D., F. Hohne, X. Li, et al. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. In 34th Advances in Neural Information Processing Systems. 2021\n\nLIMITATIONS:\nYes, the authors carefully address them.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a scalable decoupled model designed to address the challenges posed by large-scale heterophilous graph. \n\nThe model consists of two main components. In the first component, recognizing the effectiveness of $A^2$ in heterophilious graph, the model precomputes a matrix, denoted as $P_A$ in Equation (2), which serves as a representation of graph structure. This matrix is computed using a randomized eigendecomposition technique. In the second component, the authors leverage multiple hop-based coefficients of propagation to address the issue of long-range dependency. These coefficients are designed to capture information from nodes at different distances in the graph, allowing the model to effectively capture dependencies that span across multiple hops. Finally, the two components are combined, and the resulting representation is passed through an MLP to generate the final node representations. \n\nThe authors provide some theoretical support and conduct a complexity analysis to demonstrate the effectiveness and efficiency of the proposed model. The author also conducts extensive experiments to evaluate the performance of their model on heterophilious graphs with different scales. The model achieves outstanding performance compared to some popular baselines while minimizing the computational resources (GPU) required.\n\nSTRENGTHS:\n1. The authors employ a technique to compress the graph structure $A^2$ into a low-rank format while preserving the principal components. This approach effectively reduces the computational cost associated with the training phase. \n2. The authors conduct various experiments to investigate the coefficients of propagation and successfully identify the optimal approach for handling heterophilous graphs.\n3. The theoretical time complexity of the model is excellent. The pre-computation phase is conducted once, and the training phase supports mini-batch processing. These properties guarantee the scalability of the model.\n4. The performance of the model surpasses the majority of the existing results, thereby confirming its effectiveness.\n\nWEAKNESSES:\n1. The contributions to address heterophily are limited as the involvement of the raw structure of a graph and multi-hop neighbors have already been proposed in LINKX and spectral GNNs.\n2. The method employed to reduce the dimensionality of the adjacency matrix, randomized SVD, is a commonly used approach. However, its performance heavily relies on the number of iterative rounds. The authors appear to merge the computational cost of SVD associated with the propagation step, which could potentially limit the effectiveness of the algorithm.\n3. The experiments conducted in this work exclude results from well-known homophilous datasets such as Cora, PubMed, and Ogbn-products. Since this model is specifically designed for heterophilous graphs, it should still maintain comparable results on homophilous datasets when compared to the mentioned baselines.\n\nQUESTIONS:\n1. What is the performance of the model on homophilous datasets? Is the performance comparable to some recent works?\n\n2. Will the model be implemented ordinarily, particularly when confronted with extremely large graphs such as Ogbn-Papers100M and real-world data with extremely high dimensions of node embedding?\n\n3. It is mentioned in the paper [1] that there are additional heterophilous datasets available. Therefore, conducting more comprehensive experiments on those datasets is necessary to strengthen the evaluation and analysis presented in this work.\n\n[1] Platonov O, Kuznedelev D, Diskin M, et al. A critical look at the evaluation of GNNs under heterophily: are we really making progress? ICLR, 2023.\n\nLIMITATIONS:\nThe Effectiveness of randomized SVD when the propagation steps are limited is doubted.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies an important problem of graph learning on large-scale heterophily graphs.\nThe paper presents a novel approach LD2 model, decouples the embedding process from the convolutional process, allowing for more efficient and scalable learning.\nLD2 learns graphs under heterophily, which is particularly useful for large-scale graphs. \nExtensive experiments demonstrate the effectiveness of the proposed method compared to existing baselines.\n\nSTRENGTHS:\n- The paper addresses an important problem on large-scale heterophily graphs, which is especially important for large-scale datasets.\n- This paper propose a scalable graph learning method LD2 model, which decouples the embedding process from the convolutional process, and allows for more efficient and scalable learning.\n- The scalability problem is well defined, and the theoretical comparisons with previous works is clearly clarified. \n- Sufficient experiments show that LD2 is capable of lightweight minibatch training on large-scale heterophilous graphs, with up to 15× speed improvement and efficient memory utilization, while maintaining comparable or better performance than the baselines. \n- Experiments on several benchmark datasets demonstrate the effectiveness of the LD2 model compared to existing baselines, and the results show that the LD2 model outperforms existing methods in terms of both accuracy and scalability.\n\nWEAKNESSES:\n- The paper could benefit from a more detailed explanation of the LD2 model. While the authors provide some high-level descriptions of the model, it would be helpful to have a more in-depth explanation of the underlying mechanisms.\n- It is difficult to reproduce the results without access to the code used in the experiments. While the authors provide some details on the experimental setup, it would be helpful to have access to the code to ensure that the results are reproducible.\n\nI am willing to increase the score for this paper if my major concern about the reproducible problem is addressed.\n\nQUESTIONS:\n- Is it possible for the authors to make the code used in the experiments public? This would help ensure that the results are reproducible and would be a valuable resource for researchers interested in implementing the LD2 model.\n - Could the authors provide more detailed information on the underlying mechanisms of the LD2 model?\n- Could the LD2 model handle noisy or incomplete graph data? While the authors mention that the approach is robust to noise, it would be helpful to have a more detailed analysis of the model's performance under different levels of noise and incompleteness.\n\nLIMITATIONS:\nLimitations and potential improvements are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new graph neural network (GNN) model called LD2, which specifically targets learning on heterophilous graphs, where connected nodes have different labels. The authors argue that existing models for heterophilous graphs often require iterative full-graph computations, which can be computationally expensive and difficult to scale to larger graphs. In contrast, LD2 decouples graph propagation and generates expressive embeddings prior to training, resulting in a scalable and efficient model with optimal time complexity and a memory footprint that remains independent of the graph scale.\n\nSTRENGTHS:\n1. The paper studies the scalability issues of heterophilous GNN and propose a scalable model, LD2 which simplifies the learning process by decoupling graph propagation and generating expressive embeddings prior to training.\n\n2. Theoretical analysis demonstrates that LD2 achieves optimal time complexity in training, as well as a memory footprint that remains independent of the graph scale.\n\n3. Extensive experiments to showcase that the proposed model is capable of lightweight minibatch training on large-scale heterophilous graphs, with up to 15× speed improvement and efficient memory utilization, while maintaining comparable or better performance than the baselines.\n\n4. The paper is well-written and easy to follow.\n\nWEAKNESSES:\nSee questions.\n\nQUESTIONS:\n1. It seems that the method has to precompute the embeddings for adajcencies, e.g. eigendecomposition in Section 3.2, is the model able to tackle the inductive settings?\n\n2. Can other methods for encoding structures act as adajcency embeddings? for example, the spatial encodings in Graphormer.\n\n3. In line 160, it says \"Particularly, the most informative aspects are often associated with 2-hop neighbors\", is this statement verified in some paper? The method design includes the 2-hop neighborhood. What about consider other hops of neighborhood? \n\n4. Can this method apply to heterogeneous graphs?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}