Title: Faster Local Solvers for Graph Diffusion Equations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Diffusion Equation

Personalized PageRank

Heat Kernel

Katz

Local Solvers

Graph Neural Networks
Abstract: Efficient computation of graph diffusion equations (GDEs), such as Personalized PageRank, Katz centrality, and the Heat kernel, is crucial for clustering, training neural networks, and many other graph-related problems. Standard iterative methods require accessing the whole graph per iteration, making them time-consuming for large-scale graphs. While existing local solvers approximate diffusion vectors through heuristic local updates, they often operate sequentially and are typically designed for specific diffusion types, limiting their applicability. Given that diffusion vectors are highly localizable, as measured by the participation ratio, this paper introduces a novel framework for approximately solving GDEs using a local diffusion process. This framework reveals the suboptimality of existing local solvers. Furthermore, our approach effectively localizes standard iterative solvers by designing simple and provably sublinear time algorithms. These new local solvers are highly parallelizable, making them well-suited for implementation on GPUs. We demonstrate the effectiveness of our framework in quickly obtaining approximate diffusion vectors, achieving up to a hundred-fold speed improvement, and its applicability to large-scale dynamic graphs. Our framework could also facilitate more efficient local message-passing mechanisms for GNNs.

Faster Local Solvers for Graph Diffusion Equations

Abstract

Efficient computation of graph diffusion equations (GDEs), such as Personalized PageRank, Katz centrality, and the Heat kernel, is crucial for clustering, training neural networks, and many other graph-related problems. Standard iterative methods require accessing the whole graph per iteration, making them time-consuming for large-scale graphs. While existing local solvers approximate diffusion vectors through heuristic local updates, they often operate sequentially and are typically designed for specific diffusion types, limiting their applicability. Given that diffusion vectors are highly localizable, as measured by the participation ratio , this paper introduces a novel framework for approximately solving GDEs using a local diffusion process . This framework reveals the suboptimality of existing local solvers. Furthermore, our approach effectively localizes standard iterative solvers by designing simple and provably sublinear time algorithms. These new local solvers are highly parallelizable, making them well-suited for implementation on GPUs. We demonstrate the effectiveness of our framework in quickly obtaining approximate diffusion vectors, achieving up to a hundred-fold speed improvement, and its applicability to large-scale dynamic graphs. Our framework could also facilitate more efficient local message-passing mechanisms for GNNs.

1 Introduction

Graph diffusion equations (GDEs), such as Personalized PageRank (PPR) [41, 44, 62], Katz centrality (Katz) [46], Heat kernel (HK) [21], and Inverse PageRank (IPR) [51], are fundamental tools for modeling graph data. These score vectors for nodes in a graph capture various aspects of their importance or influence. They have been successfully applied to many graph learning tasks including local clustering [2, 81], detecting communities [49], semi-supervised learning [78, 80], node embeddings [63, 67], training graph neural networks (GNNs) [10, 16, 19, 31, 75], and many other applications [32]. Specifically, given a propagation matrix M associated with an undirected graph G ( V , E ) , a general graph diffusion equation is defined as

where f is the diffusion vector computed from a source vector s , and the sequence of coefficients c k satisfies c k ≥ 0 . Equation (1) represents a system of linear, constant-coefficient ordinary differential equation, ˙ x ( t ) = Mx ( t ) , with an initial condition x (0) and t ≥ 0 . Standard solvers [34, 57] for computing f require access to matrix-vector product operations Mx , typically involving O ( m ) operations, where m is the total number of edges in G . This could be time-consuming when dealing with large-scale graphs [42].

A key property of f is the high localization of its entry magnitudes, which reside in a small portion of G . Figure 1 demonstrates this localization property of f on PPR, Katz, and HK. Leveraging this locality property allows for more efficient approximation using local iterative solvers , which heuristically avoid O ( m ) operations. Local push-based methods [2, 7, 49] or their variants [4, 12] are known to be closely related to Gauss-Seidel [49, 18]. However, current local push-based methods are fundamentally sequential iterative solvers and focused on specific types such as PPR or HK. These disadvantages limit their applicability to modern GPU architectures and their generalizability.

By leveraging the locality of f , we propose, for the first time, a general local iterative framework for solving GDEs using a local diffusion process . A novel component of our framework is to model local diffusion as a locally evolving set process inspired by its stochastic counterpart [58]. We use this framework to localize commonly used standard solvers, demonstrating faster local methods for approximating f . For example, the local gradient descent is simple, provably sublinear, and highly parallelizable. It can be

6

12

18

1

2

3

4

5

7

8

9

10

11

13

14

15

16

17

Graph Index

0.05

10

-

1

10

-

3

10

-

5

10

-

7

10

-

2

10

-

4

10

-

6

10

-

8

Participation Ratio

p

(

f

)

PPR

HK

accelerated further by using local momentum. Our contributions are

By demonstrating that popular diffusion vectors, such as PPR, Katz, and HK, have strong localization properties using the participation ratio, we propose a novel graph diffusion framework via a local diffusion process for efficiently approximating GDEs. This framework effectively tracks most energy during diffusion while maintaining local computation.

To demonstrate the power of our proposed framework, we prove that APPR [2], a widely used local push-based algorithm, can be treated as a special case. We provide better diffusion-based bounds ˜ Θ(vol( S t ) / ( α · γ t )) where vol( S t ) /γ t is a lower bound of 1 /ϵ . This bound is effective for both PPR and Katz and is empirically smaller than Θ(1 / ( αϵ )) , previously well-known for APPR.

We design simple and fast local methods based on standard gradient descent for APPR and Katz, which admit runtime bounds of min(vol( S t ) / ( α · γ t ) , 1 / ( αϵ )) for both cases. These methods are GPU-friendly, and we demonstrate that this iterative solution is significantly faster on GPU architecture compared to APPR. When the propagation matrix M is symmetric, we show that this local method can be accelerated further using the local Chebyshev method for PPR and Katz.

Experimental results on GDE approximation of PPR, HK, and Katz demonstrate that these local solvers significantly accelerate their standard counterparts. We further show that they can be naturally adopted to approximate dynamic diffusion vectors. Our experiments indicate that these local methods for training are twice as fast as standard PPR-based GNNs. These results may suggest a novel local message-passing approach to train commonly used GNNs.

All proofs, detailed experimental settings, and related works are postponed to the appendix. Our code is publicly available at https://github.com/JiaheBai/Faster-Local-Solver-for-GDEs.

2 GDEs, Localization, and Existing Local Solvers

-

Notations. We consider an undirected graph G ( V , E ) where V = { 1 , 2 , . . . , n } is the set of nodes, and E is the edge set with |E| = m . The degree matrix is D = diag ( d 1 , d 2 , . . . , d n ) , and A is the adjacency matrix of G . The volume of subset nodes S is vol( S ) = ∑ v ∈S d v . The set of neighbors of node v is denoted by N ( v ) . The standard basis for node s is e s . The support of x is defined as supp( x ) = { u : x u = 0 } .

̸

Many graph learning tools can be represented as diffusion vectors. Table 1 presents examples widely used as graph learning tools. The coefficient c k usually exhibits exponential decay, so most of the energy of f is related to only the first few c k . We revisit these GDEs and existing local solvers.

Personalized PageRank. Given the weight decaying strategy c k = α (1 -α ) k and a predefined damping factor α ∈ (0 , 1) with M = AD -1 , the analytic solution for PPR is

These vectors can be used to find a local graph cut or train GNNs [8, 10, 25, 79]. The well-known approximate PPR (APPR) method [2] locally updates the estimate-residual pair ( x , r ) as follows

Here, each active node u has a large residual r u ≥ ϵαd u with initial values x ← 0 and r ← α e s . The runtime bound for reaching r u < ϵαd u for all nodes is graph-independent and is Θ(1 / ( αϵ )) , leveraging the monotonicity property. Previous analyses [49, 18] suggest that APPR is a localized version of the Gauss-Seidel (GS) iteration and thus has fundamental sequential limitations.

Katz centrality. Given the weight c k = α k +1 with α ∈ (0 , 1 / ∥ A ∥ 2 ) and a different propagation matrix M = A . The solution for Katz centrality can then be rewritten as:

The goal is to approximate the Katz centrality (( I -α A ) -1 -I ) e s . Similar to APPR, a local solver [11] for Katz centrality, denoted as AKatz, can be designed as follows

AKatz is a coordinate descent method [11]. Under the nonnegativity assumption of r (i.e., α ≤ 1 /d max ), a convergence bound for the residual ∥ r t +1 ∥ 1 is provided.

Heat Kernel. The heat kernel (HK) [21] is useful for finding smaller but more precise local graph cuts. It uses a different weight decaying c k = τ k e -τ /k ! and the vector is then defined as

where τ is the temperature parameter of the HK equation and M = AD -1 . Unlike the previous two, this equation does not admit an analytic solution. Following the technique developed in [48, 49], given an initial vector s and temperature τ , the HK vector can be approximated using the first N terms of the Taylor polynomial approximation: define x N = ∑ N k =0 v k with v 0 = e s and v k +1 = Mv k /k for k = 0 , . . . , N . It is known that ∥ f HK -x N ∥ 1 ≤ 1 / ( N ! N ) . This is equivalent to solving the linear system ( I N +1 ⊗ I n -S N +1 ⊗ M ) v = e 1 ⊗ e s , where ⊗ denotes the Kronecker product. Then, the push-based method, namely AHK, has the following updates

There are many other types of GDEs, such as Inverse PageRank [51], which generalize PPR. Many GNN propagation layers, such as SGC [69] and APPNP [30], can be formulated as GDEs. For all these f , we use the participation ratio [54] to measure its localization ability, defined as

When the entries in f are uniformly distributed, such that f i ∼ O (1 /n ) , then p ( f ) = O (1) . However, in the extremely sparse case where f is a standard basis vector, p ( f ) = 1 /n indicates the sparsity effect. Figure 1 illustrates the participation ratios, showing that almost all vectors have ratios below 0.01. Additionally, larger graphs tend to have smaller participation ratios.

APPR, AKatz, and AHK all have fundamental limitations due to their reliance on sequential GaussSeidel-style updates, which limit their parallelization ability. In the following sections, we will develop faster local methods using the local diffusion framework for solving PPR and Katz.

3 Faster Solvers via Local Diffusion Process

We introduce a local diffusion process framework, which allows standard iterative solvers to be effectively localized. By incorporating this framework, we show that the computation of PPR and Katz defined in (2) and (3) can be locally approximated sequentially or in parallel on GPUs.

3.1 Local diffusion process

To compute f PPR and f KATZ , it is equivalent to solving the linear systems, which can be written as

where Q is a (symmetric) positive definite matrix with eigenvalues bounded by µ and L , i.e., µ ≤ λ ( Q ) ≤ L . For PPR, we solve ( I -(1 -α ) AD -1 ) x = α e s , which is equivalent to solving ( I -(1 -α ) D -1 / 2 AD -1 / 2 ) D -1 / 2 x = α D -1 / 2 e s . For Katz, we compute ( I -α A ) x = e s . The vector s is sparse, with supp( s ) ≪ n . We define local diffusion process , a locally evolving set procedure inspired by its stochastic counterpart [58] as follows

Definition 3.1 (Local diffusion process) . Given an input graph G , a source distribution s , precision tolerance ϵ , and a local iterative method A with parameter θ for solving (4), the local diffusion process is defined as a process of updates { ( x ( t ) , r ( t ) , S t ) } 0 ≤ t ≤ T . Specifically, it follows the dynamic system where ϕ is a mapping via some iterative solver A θ . For all three diffusion processes (PPR, Katz, and HK), we set S 0 = { s } . We say this process converges when S T = ∅ if there exists such T ; the generated sequence of active nodes are S t . The total number of operations of the local solver A θ is

Intuitively, we can treat this local diffusion process as a random walk on a Markov Chain defined on the space of all subsets of V , where the transition probability represents the operation cost. More efficient local solvers try to find a shorter random walk from S 0 to S T . The following two subsections demonstrate the power of this process in designing local methods for PPR and Katz.

where we denote the average of active volume as vol( S T ) = T -1 ∑ T -1 t =0 vol( S t ) .

3.2 Sequential local updates via Successive Overrelaxation (SOR)

Given the estimate x ( t ) at t -th iteration, we define residual r ( t ) = s -Qx ( t ) and Q be the matrix induced by A and D . The typical local Gauss-Seidel with Successive Overrelaxation has the following online estimate-residual updates (See Section 11.2 of [34]): at each time t = 0 , 1 , . . . , T -1 , for each active node u i ∈ S t = { u 1 , u 2 , . . . , u |S t | } with i = 1 , 2 , . . . , |S t | , we update each u i -th entry of x and corresponding residual r as the following

where t i ≜ ( i -1) / |S t | is the current time, and update unit vector is ˜ e ( t + t i ) u i ≜ r ( t + t i ) u i e u i /q u i u i . Note that each update of (6) is Θ( d u i ) . If S t = V , it reduces to the standard GS-SOR [34]. Therefore, APPR is a special case of (6), a local variant of GS-SOR with ω = 1 . Figure 2 illustrates this procedure. APPR updates x at some entries and keeps track of large magnitudes of r per iteration, while LocalSOR allows for a better choice of ω ; hence, the total number of operations is reduced. We establish the following fundamental property of LocalSOR.

̸

Theorem 3.2 (Properties of local diffusion process via LocalSOR) . Let Q ≜ I -β P where P ≥ 0 n × n and P uv = 0 if ( u, v ) ∈ E ; 0 otherwise. Define maximal value P max = max u ∈V ∥ Pe u ∥ 1 . Assume that r (0) ≥ 0 is nonnegative and P max , β are such that βP max < 1 , given the updates of (6) , then the local diffusion process of ϕ ( x ( t ) , r ( t ) , S t ; s , ϵ, G , A θ = ( LocalSOR , ω ) ) with ω ∈ (0 , 1) has the following properties

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

S

0

=

{

0

}

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

S

1

=

{

1

,

12

}

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

S

2

=

{

0

,

2

,

9

,

13

,

14

}

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

S

3

=

{

1

,

12

}

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

S

4

=

{

0

,

2

,

5

,

13

}

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

S

5

=

{

14

}

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

0

=

0

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

S

1

=

{

0

,

1

,

12

}

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

S

2

=

{

1

,

2

,

9

,

12

,

14

}

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

S

3

=

{

5

,

13

}

0

1

12

2

5

9

3

6

4

7

8

11

10

13

14

S

4

=

{

4

,

12

}

S

{

}

If the local diffusion process converges (i.e., S T = ∅ ), then T is bounded by

Based on Theorem 3.2, we establish the following sublinear time bounds of LocalSOR for PPR.

Theorem 3.3 (Sublinear runtime bound of LocalSOR for PPR) . Let I T = supp( r ( T ) ) . Given an undirected graph G and a target source node s with α ∈ (0 , 1) , ω = 1 , and provided 0 < ϵ ≤ 1 /d s , the run time of LocalSOR in Equ. (6) for solving ( I -(1 -α ) AD -1 ) f PPR = α e s with the stop condition ∥ D -1 r ( T ) ∥ ∞ ≤ αϵ and initials x (0) = 0 and r (0) = α e s is bounded as the following

where C PPR = 1 / ((1 -α ) |I T | ) . The estimate x ( T ) satisfies ∥ D -1 ( x ( T ) -f PPR ) ∥ ∞ ≤ ϵ .

0

.

9

1

.

0

1

.

1

1

.

2

1

.

3

1

.

4

0

.

7

0

.

8

0

.

9

1

.

0

1

.

1

# of Operations (

∑

t

vol(

S

t

)

)

×

10

6

LocalSOR

LocalGS

ω

stop condition ∥ D -1 r ( T ) ∥ ∞ ≤ ϵ and initials x (0) = 0 and r (0) = e s is bounded as the following

The above theorem demonstrates the usefulness of our framework. It shows a new evolving bound where vol( S T ) /α T ≤ 1 /ϵ as long as Q satisfying certain assumption (It is true for PPR and Katz). Similarly, applying Theorem 3.2, we have the following result for approximating f Katz .

Corollary 3.4 (Runtime bound of LocalSOR for Katz) . Let I T = supp( r ( T ) ) and C Katz = 1 / ((1 -α ) |I T | ) . Given an undirected graph G and a target source node s with α ∈ (0 , 1 /d max ) , ω = 1 , and provided 0 < ϵ ≤ 1 /d s , the run time of LocalSOR in Equ. (6) for solving ( I -α A ) x = e s with the

Accerlation when Q is Stieltjes matrix ( ω ∗ = 2 / (1 + √ 1 -( α -1) 2 ) ). It is well-known that when ω ∈ (1 , 2] , GS-SOR has acceleration ability when Q is Stieltjes. However, it is fundamentally

The estimate ˆ f Katz = x ( T ) -e s satisfies ∥ ˆ f Katz -f Katz ∥ 2 ≤ ∥ ( I -α A ) -1 D ∥ 1 · ϵ .

difficult to prove the runtime bound as the monotonicity property no longer holds. It has been conjectured that a runtime of ˜ O (1 / ( √ αϵ )) can be achieved [27]. Incorporating our bound, a new conjecture could be ˜ O (vol( S T ) / ( √ αγ T )) : If Q is Stieltjes, then with the proper choice of ω , one can achieve speedup convergence to ˜ O (vol( S T ) / ( √ αα T )) ? We leave it as an open problem. 3.3 Parallelizable local updates via GD and Chebyshev

The fundamental limitation of LocalSOR is its reliance on essentially sequential online updates, which may be challenging to utilize with GPU-type computing resources. Interestingly, we developed a local iterative method that is embarrassingly simpler, highly parallelizable, and provably sublinear for approximating PPR and Katz. First, one can reformulate the equation of PPR and Katz as 2

A natural idea for solving the above is to use standard GD. Hence, following a similar idea of local updates, we simultaneously update solutions for S t . We propose the following local GD.

̸

Nonnegativity. r ( t ) ≥ 0 for all t ≥ 0 .

Theorem 3.5 (Properties of local diffusion process via LocalGD) . Let Q ≜ I -β P where P ≥ 0 n × n and P uv = 0 if ( u, v ) ∈ E ; 0 otherwise. Define maximal value P max = max S t ⊆V ∥ Pr S t ∥ 1 / ∥ r S t ∥ 1 . Assume that r (0) = s ≥ 0 and P max , β are such that βP max < 1 , given the updates of (10) , then the local diffusion process of ϕ ( S t , x ( t ) , r ( t ) ; G , A θ = ( LocalGD , µ, L ) ) has the following properties

If the local diffusion process converges (i.e., S T = ∅ ), then T is bounded by

Indeed, the above local updates have properties that are quite similar to those of SOR. Based on Theorem 3.5, we establish the sublinear runtime bounds of LocalGD for solving PPR and Katz.

Corollary 3.6 (Convergence of LocalGD for PPR and Katz) . Let I T = supp( r ( T ) ) and C = 1 (1 -α ) |I T | . Use LocalGD to approximate PPR or Katz by using iterative procedure (10) . Denote T PPR and T Katz as the total number of operations needed by using LocalGD, they can then be bounded by

for a stop condition ∥ D -1 r ( t ) ∥ ∞ ≤ α PPR · ϵ . For solving KATZ , then the toal runtime is bounded by

for a stop condition ∥ D -1 r ( t ) ∥ ∞ ≤ ϵd u . The estimate of equality is the same as that of LocalSOR. Remark 3.7 . Note that LocalGD is quite different from iterative hard-thresholding methods [43] where the time complexity of the thresholding operator is O ( n ) at best, hence not a sublinear algorithm.

Accerlerated local Chevbyshev. One can extend the Chebyshev method [39], an optimal iterative solver for (9). Following [24], Chebyshev polynomials T n are defined via following recursions

Given δ 1 = L -µ L + µ , x 1 = x 0 -2 L + µ ∇ f ( x 0 ) , the standard Chevyshev method is defined as

where µ ≤ λ ( Q ) ≤ L . For example, for approximating PPR, we propose the following local updates

The sublinear runtime analysis for LocalCH is complicated since it does not follow the monotonicity property during the updates. Whether it admits, an accelerated rate remains an open problem.

4 Applications to Dynamic GDEs and GNN Propagation

Our accelerated local solvers are ready for many applications. We demonstrate here that they can be incorporated to approximate dynamic GDEs and GNN propagation. We consider the discretetime dynamic graph [47] which contains a sequence of snapshots of the underlying graphs, i.e., G 0 , G 1 , . . . , G T . The transition from G t -1 to G t consists of a list of events O t involving edge deletions or insertions. The graph G t is then represented as: G 0 O 1 - - →G 1 O 2 - - →G 2 O 3 - - →···G T -1 O T - - → G T . The goal is to calculate an approximate f t for the PPR linear system Q t f t = α e s at time t . That is,

The key advantage of updating the above equation (presented in Algorithm 3) is that the APPR algorithm is used as a primary component for updating, making it much cheaper than computing from scratch. Consequently, we can apply our local solver LocalSOR to the above equation, and we found that it significantly sped up dynamic GDE calculations and dynamic PPR-based GNN training.

5 Experiments

Datasets. We conduct experiments on 18 graphs ranging from small-scale ( cora ) to large-scale ( papers100M ), mainly collected from Stanford SNAP [45] and OGB [42] (see details in Table 3). We focus on the following tasks: 1) approximating diffusion vectors f with fixed stop conditions using both CPU and GPU implementations; 2) approximating dynamic PPR using local methods and training dynamic GNN models based on InstantGNN models [75]. 3

Baselines and Experimental Setups. We consider four methods with their localized counterparts: Gauss-Seidel (GS)/LocalGS, Successive Overrelaxation (SOR)/LocalSOR, Gradient Descent (GD)/LocalGD, and Chebyshev (CH)/LocalCH. 4 Specifically, we use the stop condition ∥ D -1 r ( t ) ∥ ∞ ≤ ϵα for PPR and ∥ D -1 r ( t ) ∥ ∞ ≤ ϵ for Katz, while we follow the parameter settings used in [49] for HK. For LocalSOR, we use the optimal parameter ω ∗ suggested in Section 3.2. We randomly sample 50 nodes uniformly from lower to higher-degree nodes for all experiments. All results are averaged over these 50 nodes. We conduct experiments using Python 3.10 with CuPy and Numba on a server with 80 cores, 256GB of memory, and two 28GB NVIDIA-4090 GPUs.

5.1 Results on efficiency of local GDE solvers

Local solvers are faster and use fewer operations. Wefirst investigate the efficiency of the proposed local solvers for computing PPR and Katz centrality. We set ( α PPR = 0 . 1 , ϵ = 1 /n ) for PPR and ( α Katz = 1 / ( ∥ A ∥ 2 +1) , ϵ = 1 /m ) for Katz. We use a temperature of τ = 10 and ϵ = 1 / √ n for HK. All algorithms for HK estimate the number of iterations by considering the truncated error of Taylor approximation. Table 2 presents the speedup ratio of four local methods over their standard counterparts. For five graph datasets, the speedup is more than 100 times in terms of the number of operations in most cases. This strongly demonstrates the efficiency of these local solvers. Figure

4 presents all such results. Key observations are: 1) All local methods significantly speed up their global counterparts on all datasets. This strongly indicates that when ϵ is within a certain range, local solvers for GDEs are much cheaper than their global counterparts. 2) Among all local methods, LocalGS and LocalGD have the best overall performance. This may seem counterintuitive since LocalSOR and LocalCH are more efficient in convergence rate. However, the number of iterations needed for ϵ = 1 /n or ϵ = 1 /m is much smaller. Hence, their improvements are less significant.

6

12

18

1

2

3

4

5

7

8

9

10

11

13

14

15

16

17

Graph Index

10

4

10

6

10

8

10

10

10

5

10

7

10

9

10

11

# of Operations (

∑

t

vol(

S

t

)

)

LocalSOR

SOR

LocalGS

GS

LocalGD

LocalCH

CH

PPR

(

α

= 0

.

1

, ϵ

=

1

n

)

6

12

18

1

2

3

4

5

7

8

9

10

11

13

14

15

16

17

Graph Index

10

4

10

6

10

8

10

10

10

3

10

5

10

7

10

9

10

11

# of Operations (

∑

t

vol(

S

t

)

)

LocalSOR

SOR

LocalGS

GS

LocalGD

LocalCH

CH

(

α

=

1

1+

A

2

, ϵ

=

1

m

)

6

12

18

1

2

3

4

5

7

8

9

10

11

13

14

15

16

17

Graph Index

10

4

10

6

10

8

10

10

10

3

10

5

10

7

10

9

10

11

# of Operations (

∑

t

vol(

S

t

)

)

LocalSOR

SOR

LocalGS

GS

HK

(

τ

= 10

, ϵ

=

1

√

n

)

∥

∥

1

/n

2

-

17

2

-

20

2

-

23

2

-

26

2

-

29

2

-

32

/epsilon1

10

5

10

6

10

7

10

8

# of Operations (

∑

t

vol(

S

t

)

)

LocalSOR

LocalGS

Speedup Ratio

1

1.5

2.0

2.5

3.0

Speedup Ratio

PPR

1

/m

2

-

17

2

-

21

2

-

25

2

-

29

2

-

33

/epsilon1

10

3

10

5

10

7

10

9

# of Operations (

∑

t

vol(

S

t

)

)

LocalSOR

LocalGS

Speedup Ratio

1

2

3

4

5

6

Speedup Ratio

Which local GDE solvers are the best under what settings? In the lower precision setting, previous results suggest that LocalSOR and LocalGD perform best overall. To test the proposed LocalSOR efficiency, we use the ogbn-arxiv dataset to evaluate the local algorithm efficiency under a high precision setting. Figure 5 illustrates the performance of LocalSOR and LocalGS for PPR and Katz. As ϵ becomes smaller, the speedup of LocalSOR over LocalGS becomes more significant.

1

/n

2

-

17

2

-

20

2

-

23

2

-

26

2

-

29

2

-

32

/epsilon1

10

0

10

1

Running Time (s)

LocalGD

Speedup

Ratio

1

5

10

15

20

25

30

35

Speedup Ratio

PPR

1

/n

2

-

17

2

-

24

2

-

31

2

-

38

2

-

45

/epsilon1

10

0

10

1

Running Time (s)

LocalCH

CH

Speedup

Ratio

1

5

10

15

20

25

30

Speedup Ratio

PPR

1

/m

2

-

17

2

-

21

2

-

25

2

-

29

2

-

33

/epsilon1

10

-

1

10

0

10

1

Running Time (s)

LocalSOR

SOR

Speedup

Ratio

10

0

(

1

)

10

1

10

2

Speedup Ratio

1

/

√

n

2

-

8

2

-

12

2

-

16

2

-

20

2

-

24

/epsilon1

10

0

10

1

Running Time (s)

LocalGS

Speedup Ratio

GS

1

2

4

6

8

10

12

Speedup Ratio

HK

When do local GDE solvers (not) work? The next critical question is when standard solvers can be effectively localized, meaning under which ϵ conditions we see speedup over standard solvers. We conduct experiments to determine when local GDE solvers show speedup over their global counterparts for approximating Katz, HK, and PPR on the wiki-talk graph dataset for different ϵ values, ranging from lower precision (2 -17 ) to high precision (2 -32 ) . As illustrated in Figure 6,

when ϵ is in the lower range, local methods are much more efficient than the standard counterparts. As expected, the speedup decreases and becomes worse than the standard counterparts when high precision is needed. This indicates the broad applicability of our framework; the required precisions in many real-world graph applications are within a range where our framework is effective.

5.2 Local GDE solvers on GPU-architecture

We conducted experiments on the efficiency of the GPU implementation of LocalGD and GD, specifically using LocalGD's GPU implementation to compare with other methods. Figure 7 presents the running time of global and local solvers as a function of ϵ on the wiki-talk dataset. LocalGD is the fastest among a wide range of ϵ . This indicates that, when a GPU is available and ϵ is within the effective range, LocalGD (GPU) can be much faster than standard GD (GPU) and other methods based on CPUs. We observed similar patterns for computing Katz in Figure 11.

2

-

23

2

-

31

2

-

19

2

-

27

10

-

1

10

0

10

1

Running Time (s)

SOR

GS

GD(GPU)

LocalGD(GPU)

2

-

23

2

-

31

2

-

19

2

-

27

10

-

1

10

0

10

1

Running Time (s)

LocalGD

LocalSOR

LocalGS

GD(GPU)

LocalGD(GPU)

Citeseer

ogbn-arxiv

wiki-talk

com-orkut

wiki-en21

10

-

3

10

-

2

10

-

1

10

0

10

1

10

2

Running Time (s)

LocalSOR

SOR

LocalGS

GS

LocalGD(GPU)

GD(GPU)

LocalGD

/epsilon1

/epsilon1

5.3 Dynamic PPR approximating and training GNN models

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

0

500

1000

1500

2000

Running Time (s)

InstantGNN(LocalSOR) Propagation

InstantGNN(LocalSOR) Training

InstantGNN(LocalSOR) Inference

InstantGNN(LocalGS) Propagation

InstantGNN(LocalGS) Training

InstantGNN(LocalGS) Inference

5

10

15

0

1

2

3

4

6

7

8

9

11

12

13

14

16

67

68

69

70

71

Accuracy (%)

InstantGNN(LocalSOR)

InstantGNN(LocalGS)

Snapshot

Snapshot

5

10

15

0

1

2

3

4

6

7

8

9

11

12

13

14

16

Snapshot

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

Accumulated # of Operations(

∑

t

vol(

S

t

)

)

×

10

8

LocalSOR(Dynamic)

LocalGS(Dynamic)

LocalSOR(Static)

LocalGS(Static)

To test the applicability of the proposed local solvers, we apply LocalSOR to GNN propagation and test its efficiency on ogbn-arxiv and ogbn-products . We use the InstantGNN [75] model (essentially APPNP [30]), following the experimental settings in [75]. Specifically, we set α = 0 . 1 , ϵ = 10 -2 /n , and ω = ω ∗ to the optimal value for our cases. For ogbnarxiv , we randomly partition the graph into 16 snapshots (each snapshot with 59,375 edges), where the initial graph G 0 contains 17.9% of the edges. With a similar performance on testing accuracy, as illustrated in Figure 8, the InstantGNN model with LocalSOR (Algo. 7) has a significantly shorter training time than its local propagation counterpart (Algo. 6). Figure 9 presents the accumulated operations over these 16 snapshots. The faster local solver is LocalSOR (Dynamic), which dynamically updates ( x , r ) for (14) according to Algo. 5 and Algo. 3, whereas LocalSOR (Static) updates all approximate PPR vectors from scratch at the start of each snapshot for (14). We observed a similar pattern for LocalCH, where the number of operations is significantly reduced compared to static ones.

6 Limitations and Conclusion

When ϵ is sufficiently small, the speedup is insignificant, and it is unknown whether more efficient local solvers can be designed under this setting. Although we observed acceleration in practice, the accelerated bounds for LocalSOR and LocalCH have not been proven. Another limitation of local solvers is that they inherit the limitations of their standard counterparts. This paper mainly develops local solvers for PPR and Katz, and it remains interesting to consider other types of GDEs.

We propose the local diffusion process, a local iterative algorithm framework based on the locally evolving set process. Our framework is powerful in capturing existing local iterative solvers such as APPR for GDEs. We then demonstrate that standard iterative solvers can be effectively localized, achieving sublinear runtime complexity when monotonicity properties hold in these local solvers. Extensive experiments show that local solvers consistently speed up standard solvers by a hundredfold. We also show that these local solvers could help build faster GNN models [31, 13, 14, 20]. We expect many GNN models to benefit from these local solvers using a local message passing strategy, which we are actively investigating. Several open problems are worth exploring, such as whether these empirically accelerated local solvers admit accelerated sublinear runtime bounds without the monotonicity assumption.

Acknowledgments and Disclosure of Funding

The authors would like to thank the anonymous reviewers for their helpful comments. The work of Baojian Zhou is sponsored by Shanghai Pujiang Program (No. 22PJ1401300) and the National Natural Science Foundation of China (No. KRH2305047). The work of Deqing Yang is supported by Chinese NSF Major Research Plan No.92270121. The computations in this research were performed using the CFFF platform of Fudan University.

References

Reid Andersen, Christian Borgs, Jennifer Chayes, John Hopcraft, Vahab S Mirrokni, and ShangHua Teng. Local computation of PageRank contributions. In Algorithms and Models for the Web-Graph: 5th International Workshop, WAW 2007, San Diego, CA, USA, December 11-12, 2007. Proceedings 5 , pages 150-165. Springer, 2007.

Reid Andersen, Fan Chung, and Kevin Lang. Local graph partitioning using PageRank vectors. In 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06) , pages 475-486. IEEE, 2006.

Konstantin Avrachenkov, Paulo Gonçalves, and Marina Sokol. On the choice of kernel and labelled data in semi-supervised learning methods. In Algorithms and Models for the Web Graph: 10th International Workshop, WAW 2013, Cambridge, MA, USA, December 14-15, 2013, Proceedings 10 , pages 56-67. Springer, 2013.

Konstantin Avrachenkov, Nelly Litvak, Danil Nemirovsky, and Natalia Osipova. Monte carlo methods in PageRank computation: When one iteration is sufficient. SIAM Journal on Numerical Analysis , 45(2):890-904, 2007.

Siddhartha Banerjee and Peter Lofgren. Fast bidirectional probability estimation in markov models. Advances in Neural Information Processing Systems , 28, 2015.

Michele Benzi and Paola Boito. Matrix functions in network analysis. GAMM-Mitteilungen , 43(3):e202000012, 2020.

Pavel Berkhin. Bookmark-coloring algorithm for personalized PageRank computing. Internet Mathematics , 3(1):41-62, 2006.

Aleksandar Bojchevski, Johannes Gasteiger, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózemberczki, Michal Lukasik, and Stephan Günnemann. Scaling graph neural networks with approximate PageRank. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2464-2473, 2020.

Aleksandar Bojchevski and Stephan Günnemann. Deep Gaussian embedding of graphs: Unsupervised inductive learning via ranking. arXiv preprint arXiv:1707.03815 , 2017.

Aleksandar Bojchevski, Johannes Klicpera, Bryan Perozzi, Martin Blais, Amol Kapoor, Michal Lukasik, and Stephan Günnemann. Is PageRank all you need for scalable graph neural networks. In ACM KDD, MLG Workshop , 2019.

Francesco Bonchi, Pooya Esfandiar, David F Gleich, Chen Greif, and Laks VS Lakshmanan. Fast matrix computations for pairwise and columnwise commute times and Katz scores. Internet Mathematics , 8(1-2):73-112, 2012.

Christian Borgs, Michael Brautbar, Jennifer Chayes, and Shang-Hua Teng. A sublinear time algorithm for PageRank computations. In International Workshop on Algorithms and Models for the Web-Graph , pages 41-53. Springer, 2012.

Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. In International Conference on Machine Learning , pages 1407-1418. PMLR, 2021.

Benjamin Chamberlain, James Rowbottom, Davide Eynard, Francesco Di Giovanni, Xiaowen Dong, and Michael Bronstein. Beltrami flow and neural diffusion on graphs. Advances in Neural Information Processing Systems , 34:1594-1609, 2021.

Deli Chen, Yankai Lin, Guangxiang Zhao, Xuancheng Ren, Peng Li, Jie Zhou, and Xu Sun. Topology-imbalance learning for semi-supervised node classification. Advances in Neural Information Processing Systems , 34:29885-29897, 2021.

Ming Chen, Zhewei Wei, Bolin Ding, Yaliang Li, Ye Yuan, Xiaoyong Du, and Ji-Rong Wen. Scalable graph neural networks via bidirectional propagation. Advances in neural information processing systems , 33:14556-14566, 2020.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning , pages 1725-1735. PMLR, 2020.

Zhen Chen, Xingzhi Guo, Baojian Zhou, Deqing Yang, and Steven Skiena. Accelerating personalized PageRank vector computation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 262-273, 2023.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized PageRank graph neural network. In International Conference on Learning Representations (ICLR) , 2021.

Jeongwhan Choi, Seoyoung Hong, Noseong Park, and Sung-Bae Cho. GREAD: Graph neural reaction-diffusion networks. In International Conference on Machine Learning , pages 57225747. PMLR, 2023.

Fan Chung. The heat kernel as the PageRank of a graph. Proceedings of the National Academy of Sciences , 104(50):19735-19740, 2007.

Timothy BP Clark and Adrian Del Maestro. Moments of the inverse participation ratio for the laplacian on finite regular graphs. Journal of Physics A: Mathematical and Theoretical , 51(49):495003, 2018.

Haoran Deng, Yang Yang, Jiahe Li, Haoyang Cai, Shiliang Pu, and Weihao Jiang. Accelerating dynamic network embedding with billions of parameter updates to milliseconds. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 414-425, 2023.

Alexandre d'Aspremont, Damien Scieur, Adrien Taylor, et al. Acceleration methods. Foundations and Trends® in Optimization , 5(1-2):1-245, 2021.

Alessandro Epasto, Vahab Mirrokni, Bryan Perozzi, Anton Tsitsulin, and Peilin Zhong. Differentially private graph learning via sensitivity-bounded personalized PageRank. Advances in Neural Information Processing Systems , 35:22617-22627, 2022.

Illés J Farkas, Imre Derényi, Albert-László Barabási, and Tamas Vicsek. Spectra of 'real-world' graphs: Beyond the semicircle law. Physical Review E , 64(2):026704, 2001.

Kimon Fountoulakis and Shenghao Yang. Open problem: Running time complexity of accelerated ℓ 1 -regularized pagerank. In Conference on Learning Theory , pages 5630-5632. PMLR, 2022.

Dongqi Fu, Zhigang Hua, Yan Xie, Jin Fang, Si Zhang, Kaan Sancak, Hao Wu, Andrey Malevich, Jingrui He, and Bo Long. VCR-graphormer: A mini-batch graph transformer via virtual connections. In The Twelfth International Conference on Learning Representations , 2024.

Dongqi Fu, Dawei Zhou, and Jingrui He. Local motif clustering on time-evolving graphs. In Proceedings of the 26th ACM SIGKDD International conference on knowledge discovery & data mining , pages 390-400, 2020.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized PageRank. In International Conference on Learning Representations (ICLR) , 2019.

Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. Advances in neural information processing systems , 32, 2019.

David F Gleich. PageRank beyond the web. siam REVIEW , 57(3):321-363, 2015.

David F Gleich, Kyle Kloster, and Huda Nassar. Localization in seeded PageRank. arXiv preprint arXiv:1509.00016 , 2015.

Gene H Golub and Charles F Van Loan. Matrix computations . JHU press, 2013.

Thomas Guhr, Axel Müller-Groeling, and Hans A Weidenmüller. Random-matrix theories in quantum physics: common concepts. Physics Reports , 299(4-6):189-425, 1998.

Wentian Guo, Yuchen Li, Mo Sha, and Kian-Lee Tan. Parallel personalized PageRank on dynamic graphs. Proceedings of the VLDB Endowment , 11(1):93-106, 2017.

Xingzhi Guo, Baojian Zhou, and Steven Skiena. Subset node representation learning over large dynamic graphs. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 516-526, 2021.

Xingzhi Guo, Baojian Zhou, and Steven Skiena. Subset node anomaly tracking over large dynamic graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 475-485, 2022.

Martin H Gutknecht and Stefan Röllin. The chebyshev iteration revisited. Parallel Computing , 28(2):263-283, 2002.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Taher H Haveliwala. Topic-sensitive PageRank. In Proceedings of the 11th international conference on World Wide Web , pages 517-526, 2002.

Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogblsc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430 , 2021.

Prateek Jain, Ambuj Tewari, and Purushottam Kar. On iterative hard thresholding methods for high-dimensional m-estimation. Advances in neural information processing systems , 27, 2014.

Glen Jeh and Jennifer Widom. Scaling personalized web search. In Proceedings of the 12th international conference on World Wide Web , pages 271-279, 2003.

Leskovec Jure. Snap datasets: Stanford large network dataset collection. Retrieved December 2021 from http://snap. stanford. edu/data , 2014.

Leo Katz. A new status index derived from sociometric analysis. Psychometrika , 18(1):39-43, 1953.

Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart. Representation learning for dynamic graphs: a survey. The Journal of Machine Learning Research , 21(1):2648-2720, 2020.

Kyle Kloster and David F Gleich. A nearly-sublinear method for approximating a column of the matrix exponential for matrices from large, sparse networks. In Algorithms and Models for the Web Graph: 10th International Workshop, WAW 2013, Cambridge, MA, USA, December 14-15, 2013, Proceedings 10 , pages 68-79. Springer, 2013.

Kyle Kloster and David F Gleich. Heat kernel based community detection. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 1386-1395, 2014.

Isabel M Kloumann, Johan Ugander, and Jon Kleinberg. Block models and personalized PageRank. Proceedings of the National Academy of Sciences , 114(1):33-38, 2017.

Pan Li, I Chien, and Olgica Milenkovic. Optimizing generalized pagerank methods for seedexpansion community detection. Advances in Neural Information Processing Systems , 32, 2019.

Yiming Li, Yanyan Shen, Lei Chen, and Mingxuan Yuan. Zebra: When temporal graph neural networks meet temporal personalized PageRank. Proceedings of the VLDB Endowment , 16(6):1332-1345, 2023.

Peter Lofgren, Siddhartha Banerjee, and Ashish Goel. Bidirectional PageRank estimation: From average-case to worst-case. In Algorithms and Models for the Web Graph: 12th International Workshop, WAW 2015, Eindhoven, The Netherlands, December 10-11, 2015, Proceedings 12 , pages 164-176. Springer, 2015.

Travis Martin, Xiao Zhang, and Mark EJ Newman. Localization and centrality in networks. Physical review E , 90(5):052808, 2014.

Frank McSherry. A uniform approach to accelerated PageRank computation. In Proceedings of the 14th international conference on World Wide Web , pages 575-582, 2005.

Dingheng Mo and Siqiang Luo. Agenda: Robust personalized PageRanks in evolving graphs. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 1315-1324, 2021.

Cleve Moler and Charles Van Loan. Nineteen dubious ways to compute the exponential of a matrix, twenty-five years later. SIAM review , 45(1):3-49, 2003.

Ben Morris and Yuval Peres. Evolving sets and mixing. In Proceedings of the Thirty-Fifth Annual ACM Symposium on Theory of Computing (STOC) , page 279-286, New York, NY, USA, 2003. Association for Computing Machinery.

Huda Nassar, Kyle Kloster, and David F Gleich. Strong localization in personalized PageRank vectors. In Algorithms and Models for the Web Graph: 12th International Workshop, WAW 2015, Eindhoven, The Netherlands, December 10-11, 2015, Proceedings 12 , pages 190-202. Springer, 2015.

Huda Nassar, Kyle Kloster, and David F Gleich. Localization in seeded PageRank. Internet Mathematics , 2017.

Lorenzo Orecchia, Sushant Sachdeva, and Nisheeth K Vishnoi. Approximating the exponential, the lanczos method and an o ( m ) -time spectral algorithm for balanced separator. In Proceedings of the forty-fourth annual ACM symposium on Theory of computing , pages 1141-1160, 2012.

Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The PageRank Citation Ranking: Bringing Order to the Web. Technical report, Stanford Digital Library Technologies Project, 1998.

¸ Stefan Post˘ avaru, Anton Tsitsulin, Filipe Miguel Gonçalves de Almeida, Yingtao Tian, Silvio Lattanzi, and Bryan Perozzi. Instantembedding: Efficient local node representations. arXiv preprint arXiv:2010.06992 , 2020.

Purnamrita Sarkar, Andrew W Moore, and Amit Prakash. Fast incremental proximity search in large graphs. In Proceedings of the 25th international conference on Machine learning , pages 896-903, 2008.

Ingo Scholtes. When is a network a network? multi-order graphical model selection in pathways and temporal networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1037-1046, 2017.

Jieming Shi, Renchi Yang, Tianyuan Jin, Xiaokui Xiao, and Yin Yang. Realtime top-k personalized PageRank over large graphs on gpus. Proceedings of the VLDB Endowment , 13(1):15-28, 2019.

Anton Tsitsulin, Marina Munkhoeva, Davide Mottin, Panagiotis Karras, Ivan Oseledets, and Emmanuel Müller. FREDE: anytime graph embeddings. Proceedings of the VLDB Endowment , 14(6):1102-1110, 2021.

Hanzhi Wang, Mingguo He, Zhewei Wei, Sibo Wang, Ye Yuan, Xiaoyong Du, and Ji-Rong Wen. Approximate graph propagation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1686-1696, 2021.

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning , pages 6861-6871. PMLR, 2019.

Hao Wu, Junhao Gan, Zhewei Wei, and Rui Zhang. Unifying the global and local approaches: an efficient power iteration with forward push. In Proceedings of the 2021 International Conference on Management of Data , pages 1996-2008, 2021.

Hao Yin, Austin R Benson, Jure Leskovec, and David F Gleich. Local higher-order graph clustering. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining , pages 555-564, 2017.

Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, and Ren Chen. Decoupling the depth and scope of graph neural networks. Advances in Neural Information Processing Systems , 34:19665-19679, 2021.

Hongyang Zhang, Peter Lofgren, and Ashish Goel. Approximate personalized PageRank on dynamic graphs. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1315-1324, 2016.

Jialin Zhao, Yuxiao Dong, Ming Ding, Evgeny Kharlamov, and Jie Tang. Adaptive diffusion in graph neural networks. Advances in neural information processing systems , 34:23321-23333, 2021.

Yanping Zheng, Hanzhi Wang, Zhewei Wei, Jiajun Liu, and Sibo Wang. Instant graph neural networks for dynamic graphs. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining , pages 2605-2615, 2022.

Baojian Zhou, Yifan Sun, and Reza Babanezhad Harikandeh. Fast online node labeling for very large graphs. In International Conference on Machine Learning , pages 42658-42697. PMLR, 2023.

Baojian Zhou, Yifan Sun, Reza Babanezhad Harikandeh, Xingzhi Guo, Deqing Yang, and Yanghua Xiao. Iterative methods via locally evolving set process. arXiv preprint arXiv:2410.15020 , 2024.

Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems , 16, 2003.

Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust GNNs: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems , 34:27965-27977, 2021.

Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using Gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03) , pages 912-919, 2003.

Zeyuan Allen Zhu, Silvio Lattanzi, and Vahab Mirrokni. A local algorithm for finding wellconnected clusters. In International Conference on Machine Learning , pages 396-404. PMLR, 2013.

A Related Work

Disclaimer : The local diffusion process introduced in this paper shares a similar conceptual foundation with our concurrent work [77], where we propose the locally evolving set process. However, these two approaches address distinct problems. In this work, our focus is on investigating a different class of equations and exploring whether the local diffusion process can accelerate the training of Graph Neural Networks (GNNs).

Graph diffusion equations (GDEs) and localization. Graph diffusion vectors can either be exactly represented as a linear system or be approximations of an underlying linear system. The general form defined in Equation (1) has appeared in various literature [49, 51] (see more examples in [68]) or can be found in the survey works of [32]. The locality and localization properties of the diffusion vector f , including the PPR vector, have been explored in [64, 2, 59, 33, 60]. Inspired by methods for measuring the dynamic localization of quantum chaos [35, 22], which are also used for analyzing graph eigenfunctions [11, 26, 54], we use the inverse participation ratio to measure the localization ability of f . However, there is a lack of a local framework for solving GDEs efficiently.

Standard solvers for GDEs. The computation of a graph diffusion equation approximates the exponential of a propagation matrix, either through an inherently Neumann series or as an analytic system of linear equations. Well-established iterative methods, such as those approximating via the Neumann series or Padé approximations, exist (see more details in the surveys [57, 6, 61] and [34]). Each iteration requires computing the matrix-vector dot product, requiring O ( m ) operations for all the above methods. This is the computation we aim to avoid.

Local methods for GDEs. The most well-known local solver for solving PPR was initially proposed in [2] for local clustering. PPR, as a fundamental tool for graph learning, has been applied to many core problems, including graph neural networks [8, 17, 69, 15], graph diffusion [31, 74], and its generalizations [51, 19] for GNNs or clustering [50, 71]. Many variants have been proposed for solving such equations in different forms using local methods that couple local methods with Monte Carlo sampling strategies, achieving sublinear time [5, 53]. The work of [55] attempts to accelerate PPR computation via the Gauss-Southwell method, but it does not work well in practice [48]. A notable finding from [70] is that the local computation of PPR is equivalent to power iteration when the precision is high. There are also works on temporal PPR vector computation [66, 36, 56, 52], and our work can be immediately applied to these. Some local variants of APPR have been proposed for Katz and HK [11, 49]. However, these local solvers are sequential and lack the ability to be easily implemented on GPUs. Our local framework addresses this challenge.

PPR on dynamic graphs. Ranking on dynamic graphs has many applications [65], including motif clustering [71, 29], embedding on large-scale graphs [63, 23, 37, 38, 76, 18], and designing graph neural networks [75, 72]. The local computation via forward push (a.k.a. APPR algorithm [2]) or reverse push (a.k.a. reverse APPR [1]) is widely used in GNN propagation and bidirectional propagation algorithms [16, 28]. Our framework helps to design more efficient dynamic GNNs.

B Missing Proofs

B.1 Notations

We list all notations used in our proofs here.

e u : The indicator vector where the u -th entry is 1, and 0 otherwise.

N ( u ) : Set of neighbors of node u .

d u : The degree of node u .

G ( V , E ) : The underlying graph with the set of nodes V and set of edges E .

D : Degree matrix of G when G is undirected.

A : Adjacency matrix of G .

Π : Personalized PPR matrix, defined as Π = α ( I -(1 -α ) AD -1 ) -1 .

α Katz: Parameter α Katz ∈ (0 , 1) for the Katz equation.

α PPR: Damping factor α PPR ∈ (0 , 1) for the PPR equation.

ϵ : The error tolerance.

∥ D -1 r ∥ ∞ ≤ α PPR ϵ : The stop condition for local solvers of PPR.

∥ D -1 r ∥ ∞ ≤ ϵ : The stop condition for local solvers of Katz.

B.2 Formulation of PPR and Katz

In this section, we restate all theorems and present all missing proofs. We first restate the two target linear systems we aim to solve, as introduced in Section 3. Recall PPR is defined as ( I -(1 -α PPR ) AD -1 ) f PPR = α PPR e s . It can be equivalently written as where α PPR ∈ (0 , 1) and x ∗ = α Q -1 D -1 / 2 e s . Hence, f PPR = D 1 / 2 x ∗ . The Katz centrality can be written as

where α Katz ∈ (0 , 1 / ∥ A ∥ 2 ) . Then f Katz = x -e s .

Proposition B.1 ([3, 53]) . Let G ( V , E ) be an undirected graph and u and v be two vertices in V . Denote f s := α ( I -(1 -α ) AD -1 ) e s as the PPR vector of a source node s . Then

Proof. We directly follow the proof strategy in [53] for completeness. For path P = { s, v 1 , v 2 , . . . , v k , t } in G , we denote its length as ℓ ( P ) (here ℓ ( P ) = k + 1 ), and define its reverse path to be ¯ P = { t, v k , . . . , v 2 , v 1 , s } -note that ℓ ( P ) = ℓ ( ¯ P ) . Moreover, we know that a random walk starting from s traverses path P with probability P [ P ] = 1 d s · 1 d v 1 · · · · · 1 d v k , and thus, it is easy to see that we have

Now let P st denote the paths in G starting at s and terminating at t . Then, we can re-write f s [ t ] as:

Algo 1 APPR ( G , ϵ, α, s ) [2] adopted from [8]

1: x ← 0 , r ← α e s 2: while ∃ u, r u ≥ ϵαd u do 3: r , x ← PUSH ( u, r , x ) 4: Return x 1: PUSH ( u, r , x ) : 2: ˜ r u ← r u 3: x u ← x u + ˜ r u 4: r u ← 0 5: for v ∈ N ( u ) do 6: r v ← r v + (1 - α )˜ r u d u 7: Return ( r , x )

We adopt the typical implementation of APPR as presented in Algo. 1.

B.3 Missing proofs of LocalSOR

̸

Theorem 3.2 (Properties of local diffusion process via LocalSOR) . Let Q ≜ I -β P where P ≥ 0 n × n and P uv = 0 if ( u, v ) ∈ E ; 0 otherwise. Define maximal value P max = max u ∈V ∥ Pe u ∥ 1 . Assume that r (0) ≥ 0 is nonnegative and P max , β are such that βP max < 1 , given the updates of (6) , then the local diffusion process of ϕ ( x ( t ) , r ( t ) , S t ; s , ϵ, G , A θ = ( LocalSOR , ω ) ) with ω ∈ (0 , 1) has the following properties

Nonnegativity. r ( t + t i ) ≥ 0 for all t ≥ 0 and t i = ( i -1) / |S t | with i = 1 , 2 , . . . , |S t | .

If the local diffusion process converges (i.e., S T = ∅ ), then T is bounded by

Proof. At each step t and t i = ( i -1) / |S t | where i = 1 , 2 , . . . , |S t | , recall LocalSOR for solving Qx = s has the following updates

Since we define Q as Q = I -β P , where P uu = 0 for all u ∈ V , and using q u i u i = 1 , we continue to have the updates of r as follows

To make an effective reduction, β should be such that β ∥ Pe u i ∥ 1 < 1 for all u i ∈ V . Summing over i = 1 , 2 , . . . , |S t | of the above equation, we then have the following

By using induction, we show the nonnegativity of r ( t ) . First of all, r (0) ≥ 0 by our assumption. Let us assume r ( t + t i ) ≥ 0 , then the above equation gives r ( t + t i +1 ) ≥ 0 since ω ≤ 1 and P ≥ 0 n × n by our assumption. Since we assume ω ≤ 1 and P ≥ 0 n × n , the above equation can be written as r ( t + t i +1 ) + ω · r ( t + t i ) u i e u i = r ( t + t i ) + ωβr ( t + t i ) u i · Pe u i . By taking ∥ · ∥ 1 on both sides, we have ∥ r ( t + t i +1 ) ∥ 1 + ωr ( t + t i ) u i = ∥ r ( t + t i ) ∥ 1 + ωβr ( t + t i ) u i ∥ Pe u i ∥ 1

Note that we defined γ t = ∑ |S t | i =1 r ( t + t i ) u i ∥ r ( t ) ∥ 1 and (1 -βP max) ≤ (1 -βP u i ) for all u i ∈ S t . The inequality of (17) leads to the following bound

To further simply the above upper bound, since each term 1 -ωγ t (1 -βP max ) ≥ 0 during the updates, it follows that ωγ t (1 -βP max ) ∈ (0 , 1) . If there exists T such that S T = ∅ , then we can obtain an upper bound of T as

which leads to

Theorem 3.3 (Sublinear runtime bound of LocalSOR for PPR) . Let I T = supp( r ( T ) ) . Given an undirected graph G and a target source node s with α ∈ (0 , 1) , ω = 1 , and provided 0 < ϵ ≤ 1 /d s , the run time of LocalSOR in Equ. (6) for solving ( I -(1 -α ) AD -1 ) f PPR = α e s with the stop condition ∥ D -1 r ( T ) ∥ ∞ ≤ αϵ and initials x (0) = 0 and r (0) = α e s is bounded as the following

where C PPR = 1 / ((1 -α ) |I T | ) . The estimate x ( T ) satisfies ∥ D -1 ( x ( T ) -f PPR ) ∥ ∞ ≤ ϵ .

Proof. Recall S t = { u 1 , u 2 , . . . , u |S t | } be the set of active nodes processed in t iteration. For convenience, we denote |S t | = k . By LocalSOR defined in (6) for solving

We have the following online iteration updates for each active node u i ∈ S t (recall q u i u i = 1 ).

Note for each active node u i , we have r ( t + i -1 k ) u i ≥ ϵαd u i . The total operations for LOCSOR is

where the last equality follows from ω = 1 and ∥ r (0) ∥ 1 = α . Next, we focus on the proof of our newly derived bound ˜ O (vol( S T ) / ( αγT )) . To check the upper bound of T , from Theorem 3.2 by noticing β = (1 -α ) and P max = 1 , this leads to the following

̸

Next, we try to give a lower bound of r ( T ) . Note that after the last iteration T , for each nonzero residual r ( T ) u = 0 , u ∈ I T , there is at least one possible update that happened at node u : 1) Node u has a neighbor v u ∈ N ( u ) , which was active. This neighbor v u pushed some residual (1 -α ) r ( t ′ ) v u /d v u to u where the time t ′ < T . Hence, for all u ∈ I T , we have where the first equality is due to the nonnegativity of r guaranteeing by Theorem 3.2 and the second inequality is due to the fact that r ( t ′ ) u was active residuals before the push operation. Applying the above lower bound of ∥ r ( T ) ∥ 1 , we obtain

where C PPR = 1 / ((1 -α ) |I T | ) and I T = supp( r ( T ) ) . Combine the above inequality and the upper bound T , we obtain our new local diffusion-based bounds. The rest is to show a lower bound of 1 /ϵ . By using the active node condition, for u i ∈ S t , we have

We continue to have where the first inequality follows from the fact that ∑ |S t | i =1 r ( t + t i ) u i ∥ r (0) ∥ 1 ≤ ∑ i =1 |S t | r ( t + t i ) u i ∥ r ( t ) ∥ 1 for t = 0 , 1 , . . . , T -1 , due to the monotonicity property of ∥ r ( t ) ∥ 1 . The last inequality is from (19). Combining these, we finish the proof of the sublinear runtime bound. The rest is to prove the estimation quality. Note ∥ D -1 r ( T ) ∥ ∞ ≤ αϵ directly implies ∥ D -1 ( x ( T ) -f PPR ) ∥ ∞ ≤ ϵ by using Proposition B.1 and the corresponding stop condition.

Remark B.2 . The above theorem shares the proof strategy we provided in [77]. Here, we use a slightly different formulation of the linear system.

Corollary 3.4 (Runtime bound of LocalSOR for Katz) . Let I T = supp( r ( T ) ) and C Katz = 1 / ((1 -α ) |I T | ) . Given an undirected graph G and a target source node s with α ∈ (0 , 1 /d max ) , ω = 1 , and provided 0 < ϵ ≤ 1 /d s , the run time of LocalSOR in Equ. (6) for solving ( I -α A ) x = e s with the stop condition ∥ D -1 r ( T ) ∥ ∞ ≤ ϵ and initials x (0) = 0 and r (0) = e s is bounded as the following

Proof. The linear system ( I -α A ) x = s using LocalSOR updates can be written as

where q = 1 . The updates of r i can be simplified as the following

The proof of nonnegativity of r ( t ) follows the similar induction as in previous theorem by noticing the spectral radius of A is | λ ( A ) | ≤ d max . Hence, we have u i u i ( t + t ) -i i

Move the negative term to left and take ℓ 1 -norm on both sides, we have

It leads to the following

Since each r ( t + t i ) u i ≥ ϵd u i , vol( S t ) ≤ |S t | i =1 r ( t + t i ) u i /ϵ , we continue have

Finally, the above inequality gives the following upper runtime bound as

By letting β = α and P = A with P max = d max , we apply Theorem 3.2, to obtain the local diffusion-based bound as

Using the similar technique provided in the previous case, and letting ω = 1 we can have

To check the estimate equality, we have

Let ˆ f Katz := x ( T ) -e s This leads to

B.4 Missing proofs of LocalGD

̸

Nonnegativity. r ( t ) ≥ 0 for all t ≥ 0 .

Theorem 3.5 (Properties of local diffusion process via LocalGD) . Let Q ≜ I -β P where P ≥ 0 n × n and P uv = 0 if ( u, v ) ∈ E ; 0 otherwise. Define maximal value P max = max S t ⊆V ∥ Pr S t ∥ 1 / ∥ r S t ∥ 1 . Assume that r (0) = s ≥ 0 and P max , β are such that βP max < 1 , given the updates of (10) , then the local diffusion process of ϕ ( S t , x ( t ) , r ( t ) ; G , A θ = ( LocalGD , µ, L ) ) has the following properties

Monotonicity property. ∥ r (0) ∥ 1 ≥ · · · ∥ r ( t ) ∥ 1 ≥ ∥ r ( t +1) ∥ 1 · · · .

If the local diffusion process converges (i.e., S T = ∅ ), then T is bounded by

Proof. At each step t , recall LocalGD for solving Qx = ( I -β P ) x = s has the following updates

Since Q = I -β P , we continue to have the updates of r as the following

By using induction, we show the nonnegativity of r ( t ) . First of all, r (0) ≥ 0 by our assumption. Let us assume r ( t ) ≥ 0 . Then, the above equation gives r ( t +1) ≥ 0 since β ≥ 0 and P ≥ 0 n × n by our assumption. The above equation can be written as

Taking ∥ · ∥ 1 on both sides, we have

To make an effective reduction, β should be such that β ∥ Pr ( t ) S t ∥ 1 < ∥ r ( t ) S t ∥ 1 for all S t ⊆ V . For this, we then have the following

where note we defined γ t = ∥ r ( t ) S t ∥ 1 ∥ r ( t ) ∥ 1 and assumed ∥ Pr ( t ) S t ∥ 1 ≤ P max ∥ r ( t ) S t ∥ 1 . Apply (21) from t = 0 to T , it leads to the following bound

Note each of the term 1 -γ t (1 -βP max ) ≥ 0 during the updates, then γ t (1 -βP max ) ∈ (0 , 1) . To check the upper bound of T , we have

Corollary 3.6 (Convergence of LocalGD for PPR and Katz) . Let I T = supp( r ( T ) ) and C = 1 (1 -α ) |I T | . Use LocalGD to approximate PPR or Katz by using iterative procedure (10) . Denote T PPR and T Katz as the total number of operations needed by using LocalGD, they can then be bounded by

for a stop condition ∥ D -1 r ( t ) ∥ ∞ ≤ α PPR · ϵ . For solving KATZ , then the toal runtime is bounded by

for a stop condition ∥ D -1 r ( t ) ∥ ∞ ≤ ϵd u . The estimate equality is the same as of LocalSOR.

Proof. We first show graph-independent bound O (1 / ( αϵ )) for PPR computation. Since r ( t +1) = r ( t ) -Qr ( t ) S t , then rearrange it to 6

Note that entries in r ( t ) are nonnegative. It leads to

which leads to

At step t , LocalGD accesses the indices in S t . By the active node condition ϵαd u i ≤ r ( t ) u i , we have

Then the total run time of LocalGD is

where the last inequality is due to ∥ r ( t ) ∥ 1 = α . Therefore, the total run time is O (1 / ( αϵ )) . Follow the similar technique, we have sublinear runtime bound for Katz, i.e., T Kata ≤ 1 (1 -α Katz · d max ) ϵ . Next, we show the local diffusion bound for PPR. Recall LocalGD has the initial x (0) = 0 , r (0) = α e s and the following updates

Since β = 1 -α and P = AD -1 , by applying Theorem 3.5, we have the upper bound of T

Note that each nonzero r ( T ) u has at least part of the magnitude from the push operation of an active node. This means each nonzero of r ( T ) satisfies

where ˜ t ≤ T . Hence, we have

where C = 1 (1 -α ) |I T | . To see the additive error, note that r ( T ) := b -Qx ( T ) = α e s -( I -(1 -α ) AD -1 ) x ( T ) . It has the following equality

To estimate the bound | f s [ u ] -x ( T ) u | , we need to know ( ( I -(1 -α ) AD -1 ) -1 r ( T ) ) u . Specifically, the u -th entry of the above is

By Proposition B.1, we know d u f u [ v ] = d v f v [ u ] , then we continue to have

where the first inequality is due to r ( T ) v < ϵαd v , and the last equality follows from ∑ v ∈V f u [ v ] = 1 . Combining all inequalities for u ∈ V , we have the estimate | D -1 ( ˆ f -f PPR ) | 1 ≤ ϵ . We omit the proof of the sublinear runtime bound of LocalGD for Katz, as it largely follows the proof technique in Corollary 3.4. For the two lower bounds of 1 /ϵ , i.e., vol( S T ) /γ T ≤ 1 ϵ , it directly follows from the monotonicity and nonnegativity properties stated in Theorem 3.2 and Theorem 3.5.

Remark B.3 . The part of the theorem shares the similar proof strategy we provided in [77]. Here, we use a slightly different formulation of the linear system.

B.5 Implementation Details

Chebyshev for PPR. Let Q = I -(1 -α ) D -1 / 2 AD -1 / 2 and b = α D -1 / 2 e s . Then f ( t ) = D 1 / 2 x ( t ) . The eigenvalue of Q is in range [ α, 2 -α ] . So, we let L = 2 -α and µ = α .

When t = 1 , we have

δ 1 = 1 -α, x (0) = 0 , D 1 / 2 r (0) = α e s .

When t ≥ 1 , we have

For LocalCH, we change the update ˆ f ( t ) to the update ˆ f ( t ) S t . The final estimate ˆ f := D 1 / 2 x ( T ) .

Chebyshev for Katz. We want to solve ( I -α A ) x = e s . We assume µ = 1 -αλ max ( A ) , L = 1 -αλ min ( A ) . Then, the updates of LocalCH for Katz centrality are

When t = 0 , x (0) = 0 , r (0) = e s . To obtain an initial value δ 1 , we have δ 1 = αd max 7

2. When t = 1 , we have

When t ≥ 1 , it updates the estimate-residual pair as

For LocalCH, we change the update ˆ x ( t ) to the update ˆ x ( t ) S t . The final estimate is then ˆ f := ˆ x ( T ) -e s .

We omit the details of LocalSOR and LocalGD for PPR and Katz as they can directly follow from (6) and (10), respectively. We implement all our proposed local solvers via the FIFO Queue data structure.

B.6 FIFO-QUEUE and PRIORITY-QUEUE

We examine the different data structures of local methods and explore two types: the First-In-FirstOut (FIFO) Queue, which requires constant time O (1) for updates, and the Priority Queue, which is used to implement the Gauss-Southwell algorithm as described in [48, 7]. We test the number of operations needed using these two data structures on five small graphs, including CORA ( n = 19793 , m = 126842) , CORA-ML ( n = 2995 , m = 16316) , CITESEER ( n = 4230 , m = 10674) , DBLP ( n = 17716 , m = 105734) , and PUBMED ( n = 19717 , m = 88648) as used in [9] and downloaded from https://github.com/abojchevski/graph2gauss .

Figure 10 presents the ratio of the total number of operations needed by APPR-FIFO and APPRPriority-Queue. We tried four different settings; the performance of one does not dominate the other, and the Priority Queue is suitable for some nodes but not all nodes.

0

20

40

60

80

100

Source node

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

log

10

T

FIFO-Queue

log

10

T

Priority-Queue

α

=0.2,

/epsilon1

= 1

/m

Dblp

Pubmed

Citeseer

Cora-ML

0

20

40

60

80

100

Source node

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

α

=0.2,

/epsilon1

= 1

/n

Dblp

Pubmed

Citeseer

Cora-ML

0

20

40

60

80

100

Source node

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

α

=0.1,

/epsilon1

= 1

/m

Dblp

Pubmed

Citeseer

Cora-ML

0

20

40

60

80

100

Source node

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

α

=0.1,

/epsilon1

= 1

/n

Dblp

Pubmed

Citeseer

Cora-ML

̸

B.7 LocalGS(Dynamic) and LocalSOR(Dynamic)

The dynamic PPR algorithm based on LocalGS is presented in Algo. 2 and 3, initially proposed in [73]. Specifically, Algo. 2 and 3 are the components of LocalGS(Dynamic) for updating the edge event ( u, v ) while Algo. 4 and 5 are the components of LocagSOR(Dynamic) for update the edge event ( u, v ) . In practice, we follow the batch updates strategy proposed in InstantGNN [75], which we call Algo. 3 and 5 for list of edge events then call Algo. 2 and 4 once after this batch update of edge events.

Algo 2 LocalGS ( G , p s , r s , ϵ, α, s ) 1: while max u r s [ u ] ≥ ϵd out [ u ] do 2: Push( u ) 3: while min u r s [ u ] ≤ -ϵd out [ u ] do 4: Push( u ) 5: return ( p s , r s ) 6: procedure Push( u ): 7: p s [ u ] ← p s [ u ] + r s [ u ] 8: for v in N out ( u ) do 9: r s [ u ] ← 0 10: r s [ v ] ← r s [ v ] + (1 -α ) · r s [ u ] d out [ u ]

Algo 3 LOCALGS(DYNAMIC) ( G , p s , r s , ϵ, α, s, ( u, v )) 1: Apply Insert/Delete of ( u, v ) to ( p s , r s ) . 2: return LocalGS ( G , p s , r s , ϵ, α, s ) 3: procedure Insert( u, v ) 4: p s [ u ] ← p s [ u ] ∗ d out [ u ] / ( d out [ u ] -1) 5: r s [ u ] ← r s [ u ] -p s [ u ] /d out [ u ] 6: r s [ v ] ← r s [ v ] + (1 -α ) · p s [ u ] /d out [ u ] 7: procedure Delete( u, v ) 8: p s [ u ] ← p s [ u ] ∗ d out [ u ] / ( d out [ u ] + 1) 9: r s [ u ] ← r s [ u ] + p s [ u ] /d out [ u ] 10: r s [ v ] ← r s [ v ] -(1 -α ) · p s [ u ] /d out [ u ]

Algo 4 LocalSOR ( G , p s , r s , ϵ, α, s, ω ) 1: while max u r s [ u ] ≥ ϵd out [ u ] do 2: Push( u ) 3: while min u r s [ u ] ≤ -ϵd out [ u ] do 4: Push( u ) 5: return ( p s , r s ) 6: procedure Push( u ): 7: p s [ u ] ← p s [ u ] + ωr s [ u ] 8: r s [ u ] ← r s [ u ] -ωr s [ u ] 9: for v in N out ( u ) do 10: r s [ v ] ← r s [ v ]+ ω · (1 -α ) · r s [ u ] d out [ u ]

Algo 5 LOCALSOR(DYNAMIC) ( G , p s , r s , ϵ, α, ω, s, ( u, v ))

1: Apply Insert/Delete of ( u, v ) to ( p s , r s )

2: return LocalSOR

3: procedure Insert( u, v

4: p s [ u ] p s [ u ] d [ u ] / ( d [ u ]

5: r s [ u ] r s [ u ] p s [ u ] /d out [ u

6: r s [ v ] r s [ v ] + (1 α ) p s [ u ] /d [ u

7: procedure Delete( u, v )

8: p s [ u ] p s [ u ] d [ u ] / ( d [ u

9: r s [ u ] r s [ u ] + p s [ u ] /d [ u

10: r s [ v ] r s [ v ] (1 α ) p s [ u ] /d out [ u

B.8 InstantGNN(LocalGS) and InstantGNN(LcalSOR)

We present InstantGNN(LocalGS) and InstantGNN(LcalSOR) in Algo. 6 and Algo. 7, respectively. Additional parameter β is used; in practice, we choose β = 1 / 2 for all our experiments.

Algo 6 InstantGNN(LocalGS) ( , p , r , ϵ, α, s, β

G

C Datasets and More Experimental Results

C.1 Datasets Collected

We collected 18 graph datasets listed in Table 3. To test LocalSOR on GNN propagation, we use three graphs with feature data from [75].

C.2 Detailed experimental setups.

For all experiments, the maximum runtime limit is set to 14,400 seconds. All methods will terminate when this time limit is reached. Specifically, we use the following parameter settings for drawing Figure 1 and Figure 3.

Experimental settings for Figure 1. For computing PPR, we use a high precision parameter of ϵ = 10 -10 /m and set α PPR = 0 . 1 . For calculating Katz centrality, we use α Katz = 1 / ( | A | 2 +1) . The temperature for the Heat Kernel equation is set to τ = 10 .

Experimental settings Figure 3. We use the wiki-talk graph and α = 0 . 1 for PPR calculation.

Table 4 presents parameter settings of InstantGNN model training in our experiments. The GDE in InstantGNN is defined as f = ∑ ∞ k =0 α (1 -α ) k ( D β AD 1 -β ) k x . It is the same as APPNP [30] when we set β = 0 . 5 .

C.3 More experimental results

Figure 11 presents the results of GPU-implemented methods for Katz. Table 5-7 presents more details of participation ratios for PPR, Katz, and HK. The ratios in tables are without normalization. Figure 12-15 present more results of Figure 6 on different datasets and settings. Figure 16 and 17 present dynamic PPR approximating and training GNN model results on different datasets.

2

-

23

2

-

31

2

-

19

2

-

27

10

-

2

10

-

1

10

0

10

1

10

2

Running Time (s)

SOR

GS

GD(GPU)

LocalGD(GPU)

2

-

23

2

-

31

2

-

19

2

-

27

10

-

2

10

-

1

10

0

10

1

10

2

Running Time (s)

LocalGD

LocalSOR

LocalGS

GD(GPU)

LocalGD(GPU)

Citeseer

ogbn-arxiv

wiki-talk

com-orkut

wiki-en21

10

-

3

10

-

2

10

-

1

10

0

10

1

10

2

10

3

10

4

Running Time (s)

LocalSOR

SOR

LocalGS

GS

LocalGD(GPU)

GD(GPU)

LocalGD

/epsilon1

/epsilon1

A

∥

2

)

)

2

-

34

2

-

30

2

-

26

2

-

22

2

-

18

2

-

14

2

-

10

2

-

6

/epsilon1

10

-

3

6

×

10

-

4

2

×

10

-

3

3

×

10

-

3

Running Time (s)

0

5

10

15

20

25

30

35

40

Speedup Ratio

LocalSOR

SOR

Speedup

Ratio

1

/n

2

-

17

2

-

20

2

-

23

2

-

26

2

-

29

2

-

32

/epsilon1

10

-

1

10

0

Running Time (s)

LocalSOR

SOR

Speedup

Ratio

1

5

15

25

35

45

55

Speedup Ratio

2

-

28

2

-

26

2

-

24

2

-

22

2

-

20

2

-

18

2

-

16

2

-

14

/epsilon1

10

-

2

10

-

1

Running Time (s)

0

10

20

30

40

50

Speedup Ratio

LocalSOR

SOR

Speedup

Ratio

2

-

30

2

-

27

2

-

24

2

-

21

2

-

18

2

-

15

/epsilon1

10

-

1

10

0

Running Time (s)

0

5

10

15

20

25

30

Speedup Ratio

LocalSOR

SOR

Speedup

Ratio

2

-

34

2

-

30

2

-

26

2

-

22

2

-

18

2

-

14

2

-

10

2

-

6

/epsilon1

10

-

3

Running Time (s)

0

5

10

15

20

25

30

Speedup Ratio

LocalGS

GS

Speedup

Ratio

1

/n

2

-

17

2

-

20

2

-

23

2

-

26

2

-

29

2

-

32

/epsilon1

10

-

1

10

0

10

1

Running Time (s)

LocalGS

GS

Speedup

Ratio

1

5

10

15

20

25

Speedup Ratio

2

-

28

2

-

26

2

-

24

2

-

22

2

-

20

2

-

18

2

-

16

2

-

14

/epsilon1

10

-

2

10

-

1

10

0

Running Time (s)

0

10

20

30

40

50

Speedup Ratio

LocalGS

GS

Speedup

Ratio

2

-

30

2

-

27

2

-

24

2

-

21

2

-

18

2

-

15

/epsilon1

10

-

1

10

0

10

1

Running Time (s)

0

5

10

15

20

25

30

35

Speedup Ratio

LocalGS

GS

Speedup

Ratio

2

-

34

2

-

30

2

-

26

2

-

22

2

-

18

2

-

14

2

-

10

2

-

6

/epsilon1

10

-

3

10

-

2

Running Time (s)

0

5

10

15

20

25

30

35

Speedup Ratio

LocalGD

Speedup

Ratio

1

/n

2

-

17

2

-

20

2

-

23

2

-

26

2

-

29

2

-

32

/epsilon1

10

0

10

1

Running Time (s)

LocalGD

Speedup

Ratio

1

5

10

15

20

25

30

35

Speedup Ratio

2

-

28

2

-

26

2

-

24

2

-

22

2

-

20

2

-

18

2

-

16

2

-

14

/epsilon1

10

-

2

10

-

1

10

0

Running Time (s)

0

20

40

60

80

Speedup Ratio

LocalGD

Speedup

Ratio

2

-

30

2

-

27

2

-

24

2

-

21

2

-

18

2

-

15

/epsilon1

10

-

1

10

0

10

1

Running Time (s)

0

5

10

15

20

25

30

35

Speedup Ratio

LocalGD

Speedup

Ratio

2

-

34

2

-

30

2

-

26

2

-

22

2

-

18

2

-

14

2

-

10

2

-

6

/epsilon1

10

-

3

10

-

2

10

-

1

Running Time (s)

LocalCH

CH

Speedup

Ratio

0

20

40

60

80

100

120

140

160

Speedup Ratio

Citeseer

1

/n

2

-

17

2

-

24

2

-

31

2

-

38

2

-

45

/epsilon1

10

0

10

1

Running Time (s)

LocalCH

CH

Speedup

Ratio

1

5

10

15

20

25

30

Speedup Ratio

wiki-talk

2

-

39

2

-

35

2

-

31

2

-

27

2

-

23

2

-

19

2

-

15

/epsilon1

10

-

2

10

-

1

10

0

Running Time (s)

LocalCH

CH

Speedup

Ratio

0

10

20

30

40

50

60

70

Speedup Ratio

ogbn-arxiv

2

-

37

2

-

34

2

-

31

2

-

28

2

-

25

2

-

22

2

-

19

2

-

16

/epsilon1

10

-

1

10

0

10

1

Running Time (s)

LocalCH

CH

Speedup

Ratio

0

10

20

30

40

50

60

Speedup Ratio

com-youtube

2

-

35

2

-

31

2

-

27

2

-

23

2

-

19

2

-

15

2

-

11

2

-

7

/epsilon1

10

-

3

10

-

2

Running Time (s)

2

4

6

8

10

12

14

Speedup Ratio

LocalSOR

SOR

Speedup

Ratio

1

/m

2

-

17

2

-

21

2

-

25

2

-

29

2

-

33

/epsilon1

10

-

1

10

0

10

1

Running Time (s)

LocalSOR

SOR

Speedup

Ratio

10

0

(

1

)

10

1

10

2

Speedup Ratio

2

-

43

2

-

39

2

-

35

2

-

31

2

-

27

2

-

23

2

-

19

2

-

15

/epsilon1

10

-

2

10

-

1

10

0

Running Time (s)

0

20

40

60

80

Speedup Ratio

LocalSOR

SOR

Speedup

Ratio

2

-

37

2

-

33

2

-

29

2

-

25

2

-

21

2

-

17

/epsilon1

10

-

1

10

0

10

1

Running Time (s)

0

20

40

60

80

Speedup Ratio

LocalSOR

SOR

Speedup

Ratio

2

-

35

2

-

31

2

-

27

2

-

23

2

-

19

2

-

15

2

-

11

2

-

7

/epsilon1

10

-

3

10

-

2

Running Time (s)

2

4

6

8

10

12

14

Speedup Ratio

LocalGS

GS

Speedup

Ratio

2

-

33

2

-

30

2

-

27

2

-

24

2

-

21

2

-

18

/epsilon1

10

-

1

10

0

10

1

10

2

Running Time (s)

0

20

40

60

80

100

Speedup Ratio

LocalGS

GS

Speedup

Ratio

2

-

43

2

-

39

2

-

35

2

-

31

2

-

27

2

-

23

2

-

19

2

-

15

/epsilon1

10

-

2

10

-

1

10

0

10

1

Running Time (s)

0

10

20

30

40

50

Speedup Ratio

LocalGS

GS

Speedup

Ratio

2

-

37

2

-

33

2

-

29

2

-

25

2

-

21

2

-

17

/epsilon1

10

-

1

10

0

10

1

10

2

Running Time (s)

0

25

50

75

100

125

150

175

Speedup Ratio

LocalGS

GS

Speedup

Ratio

2

-

35

2

-

31

2

-

27

2

-

23

2

-

19

2

-

15

2

-

11

2

-

7

/epsilon1

10

-

3

10

-

2

Running Time (s)

0

2

4

6

8

10

12

14

Speedup Ratio

LocalGD

Speedup

Ratio

Citeseer

2

-

33

2

-

30

2

-

27

2

-

24

2

-

21

2

-

18

/epsilon1

10

0

10

1

10

2

Running Time (s)

0

25

50

75

100

125

150

175

Speedup Ratio

LocalGD

Speedup

Ratio

wiki-talk

2

-

43

2

-

39

2

-

35

2

-

31

2

-

27

2

-

23

2

-

19

2

-

15

/epsilon1

10

-

2

10

-

1

10

0

10

1

Running Time (s)

0

20

40

60

80

Speedup Ratio

LocalGD

Speedup

Ratio

ogbn-arxiv

2

-

37

2

-

33

2

-

29

2

-

25

2

-

21

2

-

17

/epsilon1

10

-

1

10

0

10

1

10

2

Running Time (s)

0

25

50

75

100

125

150

175

Speedup Ratio

LocalGD

Speedup

Ratio

com-youtube

2

-

28

2

-

25

2

-

22

2

-

19

2

-

16

2

-

13

2

-

10

2

-

7

/epsilon1

10

-

3

6

×

10

-

4

2

×

10

-

3

3

×

10

-

3

4

×

10

-

3

Running Time (s)

1

2

3

4

5

Speedup Ratio

LocalGS

GS

Speedup

Ratio

Citeseer

1

/

√

n

2

-

8

2

-

12

2

-

16

2

-

20

2

-

24

/epsilon1

10

0

10

1

Running Time (s)

LocalGS

Speedup Ratio

GS

1

2

4

6

8

10

12

Speedup Ratio

wiki-talk

2

-

30

2

-

26

2

-

22

2

-

18

2

-

14

2

-

10

2

-

6

/epsilon1

10

-

1

10

0

Running Time (s)

0

5

10

15

20

Speedup Ratio

LocalGS

GS

Speedup

Ratio

ogbn-arxiv

2

-

32

2

-

28

2

-

24

2

-

20

2

-

16

2

-

12

2

-

8

/epsilon1

10

0

10

1

Running Time (s)

0

2

4

6

8

10

Speedup Ratio

LocalGS

GS

Speedup

Ratio

com-youtube

1

/n

2

-

17

2

-

20

2

-

23

2

-

26

2

-

29

2

-

32

/epsilon1

10

0

3

×

10

-

1

4

×

10

-

1

6

×

10

-

1

2

×

10

0

# of Operations (

∑

t

vol(

S

t

)

)

LocalSOR

SOR

Speedup Ratio

1

2

3

4

5

6

Speedup Ratio

α

= 0

.

5

1

/n

2

-

17

2

-

20

2

-

23

2

-

26

2

-

29

2

-

32

/epsilon1

10

0

# of Operations (

∑

t

vol(

S

t

)

)

LocalSOR

SOR

Speedup Ratio

1

2

3

4

5

6

Speedup Ratio

α

= 0

.

2

1

/n

2

-

17

2

-

20

2

-

23

2

-

26

2

-

29

2

-

32

/epsilon1

10

-

1

10

0

# of Operations (

∑

t

vol(

S

t

)

)

LocalSOR

SOR

Speedup Ratio

1

2

3

4

5

6

Speedup Ratio

α

= 0

.

1

1

/n

2

-

17

2

-

20

2

-

23

2

-

26

2

-

29

2

-

32

/epsilon1

10

0

10

1

# of Operations (

∑

t

vol(

S

t

)

)

LocalSOR

SOR

Speedup Ratio

1

2

3

4

5

6

Speedup Ratio

α

= 0

.

05

1

/n

2

-

17

2

-

20

2

-

23

2

-

26

2

-

29

2

-

32

/epsilon1

10

0

10

1

# of Operations (

∑

t

vol(

S

t

)

)

LocalSOR

SOR

Speedup Ratio

1

2

3

4

5

6

Speedup Ratio

α

= 0

.

01

5

10

15

0

1

2

3

4

6

7

8

9

11

12

13

14

16

Snapshot

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

Accumulated # of Operations(

∑

t

vol(

S

t

)

)

×

10

8

LocalSOR(Dynamic)

LocalGS(Dynamic)

LocalSOR(Static)

LocalGS(Static)

ogbn-arxiv

5

10

15

0

1

2

3

4

6

7

8

9

11

12

13

14

Snapshot

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Accumulated # of Operations(

∑

t

vol(

S

t

)

)

×

10

9

LocalSOR(Dynamic)

LocalGS(Dynamic)

LocalSOR(Static)

LocalGS(Static)

ogbn-products

5

10

15

0

1

2

3

4

6

7

8

9

11 12 13 14

16 17 18 19 20

Snapshot

10

10

10

11

Accumulated # of Operations(

∑

t

vol(

S

t

)

)

LocalSOR(Dynamic)

LocalGS(Dynamic)

LocalSOR(Static)

LocalGS(Static)

ogbn-papers100M

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Snapshot

0

10000

20000

30000

40000

50000

60000

Running Time (s)

InstantGNN(LocalSOR) Propagation

InstantGNN(LocalSOR) Training

InstantGNN(LocalSOR) Inference

InstantGNN(LocalGS) Propagation

InstantGNN(LocalGS) Training

InstantGNN(LocalGS) Inference

5

10

15

0

1

2

3

4

6

7

8

9

11

12

13

14

Snapshot

74

75

76

77

78

79

Accuracy (%)

InstantGNN(LocalSOR)

InstantGNN(LocalGS)

C.4 The efficiency of local methods on different types of graphs (tested on grid graphs).

Our local algorithms do not depend on graph types. However, our key assumption is that diffusion vectors are highly localizable, measured by the participation ratio. As demonstrated in Figure 1, almost all diffusion vectors have low participation ratios collected from 18 real-world graphs. These graphs are diverse, ranging from citation networks and social networks to gene structure graphs. To further illustrate this, we conducted experiments where the graphs are grid graphs, and the diffusion vectors have high participation ratios (about 3 . 56 × 10 -6 , with a grid size of 1000x1000, i.e., 10 6 nodes and 1,998,000 edges). We set α PPR = 0 . 1 . The figure below presents the running time as a function of ϵ over a grid graph with 50 sampled source nodes.

2

-

12

2

-

20

2

-

28

2

-

36

2

-

44

2

-

52

/epsilon1

10

-

1

10

0

10

1

Running Time (s)

LocalGD

LocalSOR

LocalGS

SOR

GS

The results indicate that local solvers are more efficient than global ones even when ϵ is very small.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect our contributions and scope. We introduced a novel framework for approximately solving graph diffusion equations (GDEs) using a local diffusion process.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We discussed several limitations of our proposed work in the last section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper thoroughly presents theoretical results, including the full set of assumptions necessary for each theorem and lemma. Each proof is detailed and follows logically from the stated assumptions, ensuring correctness.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide comprehensive details on the experimental setup, including descriptions of the datasets used, parameter settings, and the specific algorithms applied. We also provided our code for review and will make it public upon publication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to the data and code, along with instructions in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We follow the existing experimental setup and provide detailed information on the training and testing settings used for InstantGNN with LocalSOR.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports error bars and other relevant information about the statistical significance of the experiments. We use 50 randomly sampled nodes and plot the mean and standard deviation of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The experiments were conducted using Python 3.10 with CuPy and Numba libraries on a server with 80 cores, 256GB of memory, and two NVIDIA-4090 GPUs with 28GB each.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research adheres to all aspects of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There are no broader societal impacts, as it focuses on technical contributions.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not involve data or models with a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not involve licenses for existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not involve new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing and research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve institutional review board (IRB) approvals or equivalent for research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper, \"Faster Local Solvers for Graph Diffusion Equations,\" addresses the efficiency of computing Graph Diffusion Equations (GDEs) such as Personalized PageRank (PPR), Katz centrality, and the Heat kernel, which are essential for various graph-related problems like clustering and training neural networks. Traditional methods for solving GDEs are computationally intensive for large-scale graphs. This paper introduces a novel framework for approximating GDEs using local diffusion processes, which significantly reduces computational time and improves scalability by leveraging the localization property of diffusion vectors. The proposed local solvers are highly parallelizable and suitable for GPU implementation, offering up to a hundred-fold speed improvement and applicability to large-scale dynamic graphs. The paper also discusses the potential for these methods to enhance local message-passing mechanisms in Graph Neural Networks (GNNs).\n\nSTRENGTHS:\nThe introduction of a novel framework for localizing the computation of GDEs using local diffusion processes is a significant contribution. This approach reveals the suboptimality of existing local solvers and provides a more efficient solution.\n\n\nThe paper offers a solid theoretical foundation, proving that popular diffusion vectors have strong localization properties using the participation ratio. It demonstrates that Approximate Personalized PageRank (APPR) can be treated as a special case of the proposed framework, providing better diffusion-based bounds. The design of simple and fast local methods based on standard gradient descent and the local Chebyshev method for symmetric propagation matrices is well-founded.\n\n\nExperimental results on GDE approximation for PPR, HK, and Katz show significant acceleration over standard methods. The proposed local solvers demonstrate up to a hundred-fold speed improvement. The paper also shows that the new methods can be naturally adopted to approximate dynamic diffusion vectors, and they outperform standard PPR-based GNNs in training speed.\n\nWEAKNESSES:\nPrecision Limitations: While the paper shows significant speedups for lower precision settings, the performance gain diminishes as higher precision is required. This limitation could affect the applicability of the methods in scenarios where high precision is crucial.\n\nSequential Nature: (Please correct if I am wrong.) Despite improvements, the reliance on sequential updates in some methods (like LocalSOR) still poses challenges for achieving maximal parallel efficiency.\n\nComplexity of Analysis: The runtime analysis for some of the proposed methods, such as the Local Chebyshev method, is noted as complex and remains an open problem. Maybe simplifying this analysis or providing more intuitive explanations could enhance the paper’s accessibility.\n\nQUESTIONS:\nWhat are the potential trade-offs between precision and computational efficiency, and how can practitioners balance these in practical applications?\n\nThe paper includes quite some datasets to test, with biggest of ogbn-papers100M. How does the proposed method handle very large-scale dynamic graphs in real-time applications, and what are the potential bottlenecks?\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper proposes a novel framework for approximately solving graph diffusion equations using a local diffusion process. In addition,  the proposed method can effectively localizes standard iterative solvers by designing simple and provably sublinear time algorithms.

STRENGTHS:
+ The problem is well motivated and the paper is well-written.
+ Extensive experiments are conducted to showcase the efficiency of graph diffusion framework to approximating graph diffusion equations.
+ The paper provides a good summary of existing graph diffusion equations.
+ The authors provide code and details of implementations.

WEAKNESSES:
- In this paper, the authors explore 18 different graphs, can the authors show/provide which local GDE solver achieves better performance on which type(s) of graphs? 
- Computational complexity/cost is missing.

QUESTIONS:
See comments in Weaknesses.

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
This paper proposes a new local iterative framework for solving graph diffusion equations (GDEs). Specifically, the framework approximates GDEs through a local diffusion process, leveraging the strong localization properties of diffusion vectors such as personalized PageRank, Katz centrality, and Heat Kernel. The proposed local solvers can achieve sublinear runtime complexity under certain monotonicity assumptions. Empirical results demonstrate that these solvers significantly accelerate their standard counterparts on several large-scale benchmark datasets.

STRENGTHS:
1. The paper is well-structured and clear.

2. The theoretical analysis is rigorous and sound, providing runtime complexity bounds for some of the proposed local solvers, which are better than their standard counterparts.

3. The effectiveness of the proposed methods is well-supported by experimental results on large-scale benchmark datasets.

WEAKNESSES:
1. It is not clear how graph structures, such as sparsity and spectral properties, impact the runtime complexity of the local solvers. A discussion on the potential influence of graph structures on runtime complexity would be helpful.

2. The runtime complexity analysis assumes that the updates of local solvers satisfy monotonicity properties. However, LocalCH does not satisfy these properties during updates. Establishing runtime bounds for LocalCH may require different techniques.

3. In Figure 7, when $\epsilon \geq 2^{-31}$, the running time of LocalGD is the same as GD, indicating that LocalGD may not speed up the standard counterparts when $\epsilon$ is sufficiently small.

QUESTIONS:
Please refer to Weaknesses.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a suite of fast methods to approximately compute graph diffusion vectors such as Personalized PageRank, Katz centrality and the heat kernel. A notable feature of the proposed methods is that they are easily parallelizable and hence can achieve further acceleration on GPU. The authors also provide a running time bound for each method that they introduce. Empirical results show that the new local methods can achieve up to a hundred-fold speedup when compared to their global counterpart.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow.\n- The local diffusion framework is applicable to computing several important graph diffusion vectors.\n- The experiments are reasonably comprehensive.\n\nWEAKNESSES:\n- When compared with APPR, the speedup is not really captured by Theorem 3.3 and Corollary 3.6. The authors only showed that $\\overline{\\mbox{vol}}(\\mathcal{S}_T)/\\bar\\gamma_T \\le 1/\\epsilon$, but it requires strict inequality to achieve nontrivial speedup in terms of worse-case running time. It is not clear how tight this bound is in general. If the bound is tight, then in the worst case there is no speedup. The authors should comment on if there are classes of graphs over which there will be a notable gap between the 2 quantities $\\overline{\\mbox{vol}}(\\mathcal{S}_T)/\\bar\\gamma_T$ and $1/\\epsilon$, so that the result provides a meaningful improvement.\n- Because the theorems concerning the worst-case running time do not seem to capture a clear improvement over simple baseline local methods, it is unclear where exactly the speedup reported in Table 2 comes from.\n- In LocalGD and LocalCH, the authors did not provide an update rule (or even a definition) for $\\mathcal{S}_t$. Since $\\mathcal{S}_t$ is part of the local diffusion process, the authors should specify how $\\mathcal{S}_t$ is updated or defined at each step. I guess that $\\mathcal{S}_t$ depends on the termination condition, but I could not find where the authors mention about it in the main paper.\n\nQUESTIONS:\n- Line 188: The open problem is recently solved by Martínez-Rubio et al, Accelerated and Sparse Algorithms for Approximate Personalized PageRank and Beyond, 2023.\n- When eps goes to 0, do the approximate solutions converge to the exact global solutions?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}