Title: Iterative Methods via Locally Evolving Set Process
Venue: NeurIPS 2024
Accepted: yes
Keywords: local computation

Personalized PageRank

graph clustering
Abstract: Given the damping factor $\alpha$ and precision tolerance $\epsilon$, \citet{andersen2006local} introduced Approximate Personalized PageRank (APPR), the \textit{de facto local method} for approximating the PPR vector, with runtime bounded by $\Theta(1/(\alpha\epsilon))$ independent of the graph size. Recently,  Fountoulakis \& Yang asked whether faster local algorithms could be developed using $\tilde{\mathcal{O}}(1/(\sqrt{\alpha}\epsilon))$ operations. By noticing that APPR is a local variant of Gauss-Seidel, this paper explores the question of *whether standard iterative solvers can be effectively localized*. We propose to use the *locally evolving set process*, a novel framework to characterize the algorithm locality, and demonstrate that many standard solvers can be effectively localized. Let $\overline{\operatorname{vol}}{ (\mathcal S_t)}$ and $\overline{\gamma_t}$ be the running average of volume and the residual ratio of active nodes $\textstyle \mathcal{S_t}$ during the process. We show $\overline{\operatorname{vol}}{ (\mathcal S_t)}/\overline{\gamma_t} \leq 1/\epsilon$ and prove APPR admits a new runtime bound $\tilde{\mathcal{O}}(\overline{\operatorname{vol}}(\mathcal S_t)/(\alpha\overline{\gamma_t}))$ mirroring the actual performance. Furthermore, when the geometric mean of residual reduction is $\Theta(\sqrt{\alpha})$, then there exists $c \in (0,2)$ such that the local Chebyshev method has runtime $\tilde{\mathcal{O}}(\overline{\operatorname{vol}}(\mathcal{S_t})/(\sqrt{\alpha}(2-c)))$ without the monotonicity assumption. Numerical results confirm the efficiency of this novel framework and show up to a hundredfold speedup over corresponding standard solvers on real-world graphs.

Iterative Methods via Locally Evolving Set Process

Abstract

Given the damping factor α and precision tolerance ϵ , Andersen et al. [2] introduced Approximate Personalized PageRank (APPR), the de facto local method for approximating the PPR vector, with runtime bounded by Θ(1 / ( αϵ )) independent of the graph size. Recently, Fountoulakis & Yang [12] asked whether faster local algorithms could be developed using ˜ O (1 / ( √ αϵ )) operations. By noticing that APPR is a local variant of Gauss-Seidel, this paper explores the question of whether standard iterative solvers can be effectively localized . We propose to use the locally evolving set process , a novel framework to characterize the algorithm locality, and demonstrate that many standard solvers can be effectively localized. Let vol( S t ) and γ t be the running average of volume and the residual ratio of active nodes S t during the process. We show vol( S t ) /γ t ≤ 1 /ϵ and prove APPR admits a new runtime bound ˜ O (vol( S t ) / ( αγ t )) mirroring the actual performance. Furthermore, when the geometric mean of residual reduction is Θ( √ α ) , then there exists c ∈ (0 , 2) such that the local Chebyshev method has runtime ˜ O (vol( S t ) / ( √ α (2 -c ))) without the monotonicity assumption. Numerical results confirm the efficiency of this novel framework and show up to a hundredfold speedup over corresponding standard solvers on real-world graphs.

1 Introduction

Personalized PageRank (PPR) vectors are key tools for graph problems such as clustering [2, 3, 30, 36, 54, 57], diffusion [10, 14, 15, 29], random walks [25, 32, 44], neural net training [7, 27, 20, 21], and many others [17, 48]. The Approximate PPR (APPR) [2] and its many variants [6, 9, 13, 37] efficiently approximate PPR vectors by exploring the neighbors of a specific node at each time, only requiring access to a tiny part of the graph - hence the number of operations needed is independent of graph size. These local solvers are well-suited for large-scale graphs in modern graph data analysis. Specifically, let A and D be the adjacency and degree matrices of a graph G , respectively. Given a source node s and the damping factor α ∈ (0 , 1) , this paper studies local solvers for the linear system where e s is the standard basis of s and π is the PPR vector [2, 12, 37]. Given the error tolerance ϵ , a local solver needs to find ˆ π such that ∥ D -1 (ˆ π -π ) ∥ ∞ ≤ ϵ without accessing the entire graph G . 2

Andersen et al. [2] proposed the local APPR algorithm, which pushes large residuals to neighboring nodes until all residuals are small. Its runtime is upper bounded by Θ(1 / ( αϵ )) independent of graph size. Based on a variational characterization of Equ. (1), Fountoulakis et al. [13] reformulated the problem as optimizing a quadratic objective plus ℓ 1 -regularization and later asked [12] whether there exists a local solver with runtime ˜ O (1 / ( √ αϵ )) . This corresponds to an accelerated rate since α is the strongly convex parameter. Recently, Martínez-Rubio et al. [37] provided a method based on a nested subspace pursuit strategy, and the corresponding iteration complexity is bounded by ˜ O ( |S ∗ | / √ α ) where S ∗ is the support of the optimal solution. This bound deteriorates to ˜ O ( n/ √ α ) when the solution is dense, with n representing the number of nodes in G , which could be less favorable than that of standard solvers under similar conditions. Moreover, the nested computational structure provides a constant factor overhead, which could be significant in practice.

The bound analysis of the above local methods critically depends on the monotonicity properties of the designed algorithms. These requirements may hinder the development of simpler and faster local linear solvers that lack such monotonicity properties. Specifically, the runtime analysis of APPR relies on the non-negativity and decreasing monotonicity of residuals. Conversely, the runtime bounds developed in Fountoulakis et al. [13] and Martínez-Rubio et al. [37] depend on the monotonicity of variable updates, ensuring that the sparsity of intermediate variables increases monotonically.

Our contributions. Based on a refined analysis of APPR, our starting point is to demonstrate that APPR is a local variant of Gauss-Seidel Successive Overrelaxation (GS-SOR) that can be treated as an evolving set process. 3 This insight leads us to explore whether standard solvers can be effectively localized to solve Equ. (1). To develop faster local methods with improved local bounds, we propose a novel locally evolving set process framework inspired by its stochastic counterpart [38]. This framework enables the development of faster local methods and circumvents the monotonicity requirement barrier in runtime complexity analysis in the existing literature. For example, our analysis of the local Chebyshev method does not depend on the monotonicity of residual or the active node sets processed. Specifically,

As a core tool, we propose an algorithm framework based on the locally evolving set process . We show that APPR is a local variant of GS-SOR using this process. This framework is powerful enough to facilitate the development of new local solvers. Specifically, standard gradient descent (GD) can be effectively localized for solving this problem and admits Θ(1 / ( αϵ )) runtime bound.

This local evolving set process provides a novel way to characterize the algorithm locality; hence, new runtime bounds can be derived. Let vol( S t ) and γ t be the running average of volume and the residual ratio of active nodes S t during the process; we prove the ratio vol( S t ) /γ t serving as a lower bound of 1 /ϵ . We further show both APPR and local GD have ˜ O (vol( S t ) / ( αγ t )) runtime bound mirroring the actual performance of these two methods.

Using our framework, we show there exists c ∈ (0 , 2) such that both the localized Chebyshev and Heavy-Ball methods admit runtime bound ˜ O (vol( S t ) / ( √ α (2 -c ))) with the assumption that the geometric mean of active ratio factors is Θ( √ α ) . Importantly, our analysis does not require any monotonicity property. The technical novelty is that we effectively characterize residuals of these two methods by using second-order difference equations with parameterized coefficients.

We demonstrate, over 17 large graphs, that these localized methods can significantly accelerate their standard counterparts by a large margin. Furthermore, our proposed LOCSOR, LOCCH, and LOCHB are significantly faster than APPR and ℓ 1 -based solvers on two huge-scale graphs.

Paper structure. We begin by clarifying notations and reviewing APPR in Sec. 2. Sec. 3 introduces the locally evolving set process. Sec. 4 presents localized Chebyshev and Heavy-Ball methods along with our novel techniques. We discuss open questions in Sec. 5. Experiments and conclusions are covered in Sec. 6 and 7, respectively. Detailed related works and all missing proofs are included in the Appendix. Our code is available at https://github.com/baojian/LocalCH .

2 Notations and Preliminaries

Notations. Weconsider an undirected simple graph G ( V , E ) where V = { 1 , 2 , . . . , n } and E ⊆ V×V with |E| = m are the node and edge sets, respectively. The set of neighbors of v is denoted as

2

N ( v ) ⊆ V . The adjacency matrix A of G assigns unit weight a u,v = 1 if ( u, v ) ∈ E and 0 otherwise. The v -th entry of the degree matrix D is d v = |N ( v ) | . Given S ⊆ V , we define the volume of S as vol( S ) ≜ ∑ v ∈S d v . The support of x ∈ R n is the set of nonzero indices supp( x ) ≜ { v : x v = 0 , v ∈ V} . The eigendecomposition of D -1 / 2 AD -1 / 2 = V Λ V ⊤ where each column of V is an eigenvector and Λ = diag ( λ 1 , λ 2 , . . . , λ n ) with 1 = λ 1 ≥ λ 2 ≥ · · · ≥ λ n ≥ -1 .

2.1 Revisiting Anderson's APPR and its local runtime bound

We use ( p , z ) for solving Equ. (1) while use ( x , r ) for solving Equ. (3) or Equ. (4). With the initial setting p ← 0 , z ← e s , APPR obtains a local estimate of π denoted as p by using a sequence of PUSH operations defined as

Algo. 1 PUSH ( u, α, p , z )

1:

ν

=

z

u

2:

p

u

p

u

+

α

ν

α

·

)

z

5: z v ← 6: Return ( p , z )

3: z u ← (1 -· 4: for v ∈ N ( u ) do

v

+

ν/

-

(1

α

)

2

d

u

At each repeat step ( p , z ) ← PUSH ( u, α, p , z ) , it synchronously updates both p and residual z whenever there exists an active node u ∈ V (a node with a large residual, i.e., z u ≥ ϵd u ). Specifically, for each active u , it updates p u , z u , and z v for v ∈ N ( u ) by using a PUSH operator illustrated on the left. It stops when no active nodes are left. APPR can be implemented locally so that the runtime is independent of G . In particular, vol(supp( p )) is locally bounded, demonstrating the sparsity effect. We restate the existing main results as follows.

Lemma 2.1 (Runtime bound of APPR [2]) . Given α ∈ (0 , 1) and the precision ϵ ≤ 1 /d s for node s ∈ V with p ← 0 , z ← e s at the initial, APPR ( α, ϵ, s, G ) defined in (2) returns an estimate p of π . There exists a real implementation of (2) (e.g., Algo. 2) such that the runtime T APPR satisfies

Furthermore, the estimate ˆ π := p satisfies ∥ D -1 (ˆ π -π ) ∥ ∞ ≤ ϵ and vol(supp(ˆ π )) ≤ 2 / ((1 -α ) ϵ ) .

The main argument for proving Lemma 2.1 is critically based on: 1) z ≥ 0 and ∥ z ∥ 1 decreases during the updates; 2) for each active u , z u ≥ ϵd u , implying that ∥ z ∥ 1 is decreased by at least αϵd u , consecutively leading to ∑ u d u ≤ 1 / ( αϵ ) . 2.2 Problem reformulation

To approximate the PPR vector π , the original linear system in Equ. (1) can be reformulated as an equivalent symmetric version defined as

where again e s is the standard basis of s , and Q is a symmetric positive-definite M -matrix with all eigenvalues in [ 2 α 1+ α , 2 1+ α ] . To solve Equ. (3) is equivalent to solving a quadratic problem where f is strongly convex with condition number 1 /α . Indeed, Equ. (3) is a symmetrized version of Equ. (1) and has a unique solution x ∗ = Q -1 b . The PPR vector π can be recovered from x ∗ by π = D 1 / 2 x ∗ . It is convenient to denote estimate of π as π ( t ) ≜ D 1 / 2 x ( t ) . Given x ( t ) , we define the residual r ( t ) ≜ b -Qx ( t ) . If x ( t ) is returned by a local solver for solving either Equ. (3) or Equ. (4), we then equivalently require ∥ D -1 / 2 ( x ( t ) -x ∗ ) ∥ ∞ ≤ ϵ . Hence, it is enough to have a stop condition ∥ D -1 / 2 r ( t ) ∥ ∞ ≤ 2 αϵ/ (1 + α ) for local solvers of Equ. (3) and Equ. (4). 4

Fountoulakis et al. [13] demonstrated that APPR is equivalent to a coordinate descent solver for minimizing f in Equ. (4) and introduced an ISTA-style solver by minimizing f ( x ) + ϵα ∥ D 1 / 2 x ∥ 1 , which provides a method with runtime bound ˜ O (1 / ( ϵα )) for achieving the same estimation guarantee of APPR. On one hand, one may note that the runtime bound Θ(1 / ( αϵ )) provided in Lemma 2.1 becomes less valuable when ϵ ≤ 1 /m ; on the other hand, all previous local variants [6, 9, 13, 37]

ν

̸

of APPR are critically based on some monotonicity property. This limitation could impede the development of faster local methods that might violate the monotonicity assumption. The following two sections present the techniques and tools to address these challenges.

3 Local Methods via Evolving Set Process

Our investigation begins with the locally evolving set process , as inspired by the stochastic counterpart [38]. The process reveals that APPR is essentially a local variant of GS-SOR. We then show how to use this process to build faster local solvers based on GS-SOR. We further develop a local parallelizable gradient descent with runtime Θ(1 / ( αϵ )) .

3.1 Locally evolving set process

Given α, ϵ, s , and G , a local solver for Equ. (3) keeps track of an active set S t ⊂ V at each iteration t . That is, only nodes in S t are used to update x or r . The next set S t +1 is determined by current S t and an associated local solver A . We define this process as the following local evolving set system.

Definition 3.1 (Locally evolving set process) . Given a parameter configuration θ ≜ ( α, ϵ, s, G ) , and a local iterative method A , the locally evolving set process generates a sequence of ( S t , x ( t ) , r ( t ) ) representing as the following dynamic system where S t +1 ⊆ S t ∪ ( ∪ u ∈S t N ( u )) and we denote the active set S t = { u 1 , u 2 , . . . , u |S t | } . The set S t is maintained via a queue data structure. We say this process converges when the last set S T = ∅ if there exists such T ; the generated sequence of active nodes are

The runtime of the local solver, A for this whole local process, is then defined as 5

The framework of this set process provides a new way to design local methods. Furthermore, it helps to analyze the convergence and runtime bound of local solvers by characterizing the sequences { vol( S t ) } , and {∥ r ( t ) ∥} generated by Φ θ . To analyze a new runtime bound, for T ≥ 1 , we define the average of the volume of active node sets { vol( S t ) } and active ratio sequence { γ t } as

where ∆ i is a smaller time magnitude. We define ∆ i = ( i -1) / |S t | for the analysis of APPR and LOCSOR while ∆ i = 0 for LOCGD in our later analysis. In the rest, we denote I T = supp( r ( T ) ) .

These two metrics vol( S T ) and γ T characterize the locality of local methods. To demonstrate this local process, Fig. 1 shows vol( S t ) of APPR peaks at the early stage, and the active ratio decreases as the active volume

0

20

40

t

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

×

10

5

vol(

S

t

)

vol(

S

t

)

/γt

0

20

40

t

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

γ

t

γ

t

0

20

40

t

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

×

10

6

1

//epsilon1

vol(

S

t

)

γ

t

·

ln

(

2

/epsilon1

(1

-

α

)

|I

t

|

)

diminishes. The quantity vol( S T ) /γ T is strictly smaller than 1 /ϵ , indicating that it could serve as a better factor in the runtime analysis.

3.2 APPR via locally evolving set process

We first demonstrate how this locally evolving set process can represent APPR. For solving Equ. (1), the set S 0 = { s } and the queue-based of APPR (see Algo. 2 in Appendix A) naturally forms a sequence of active sets from S 0 = { s } to S T = ∅ , hence converging. Active nodes u in queue satisfy z u ≥ ϵd u . To delineate successive iterations S t and S t +1 , one can insert ∗ at the beginning of S t . After processing S t , it serves as an indicator for the next iteration. The star ∗ is reinserted into the queue iteratively until the queue is empty. We use a slightly different notation for presenting tuple ( S t , p ( t ) , z ( t ) ) to consistent with Sec. 2.1 and write out such evolving process as follows

The following lemma establishes the equivalence between APPR and the local variant of GS-SOR method (see Appendix B.1) and provides a new evolving-based bound.

Lemma 3.2 (New local evolving-based bound for APPR) . Let M = α -1 ( I -1 -α 2 ( I + AD -1 ) ) and s = e s . The linear system Mπ = s is equivalent to Equ. (1) . Given p (0) = 0 , z (0) = e s with ω ∈ (0 , 2) , the local variant of GS-SOR (15) for Mπ = s can be formulated as

where u i is an active node in S t satisfying z u i ≥ ϵd u i and ∆ i = ( i -1) / |S t | . Furthermore, when ω = 1+ α 2 , this method reduces to APPR given in (7) , and there exists a real implementation (Aglo. 2) of APPR such that the runtime T APPR is bounded, that is

3.3 Faster local variant of GS-SOR

Lemma 3.2 points to the sub-optimality of APPR, as GS-SOR allows for a larger ω . For solving Equ. (3), since APPR essentially serves as a local variant of GS-SOR, we can develop a faster local variant based SOR. To extend this method to solve Equ. (3), we propose a local GS-SOR based on an evolving set process, namely LOCSOR, as the following

When ω ∈ (0 , 1] , the residual r is still nonnegative and monotonically decreasing, we establish the convergence of LOCSOR stated in the following theorem.

Theorem 3.3 (Runtime bound of LOCSOR ( ω = 1) ) . Given the configuration θ = ( α, ϵ, s, G ) with α ∈ (0 , 1) and ϵ ≤ 1 /d s and let r ( T ) and x ( T ) be returned by LOCSOR defined in (8) for solving Equ. (3) . There exists a real implementation of (8) such that the runtime T LOCSOR is bounded by

where vol( S T ) and γ T are defined in (6) and C = 1+ α (1 -α ) |I T | with I T = supp( r ( T ) ) . Furthermore, vol( S T ) /γ T ≤ 1 /ϵ and the local estimate ˆ π := D 1 / 2 x ( T ) satisfies ∥ D -1 (ˆ π -π ) ∥ ∞ ≤ ϵ .

Our new evolving bound ˜ O (vol( S T ) / ( αγ T )) mirroring the actual performance of APPR and empirically much smaller than Θ(1 / ( αϵ )) as illustrated in Fig. 2. Our lower bounds are quite effective when ϵ is relatively large, while our upper bound is better than Anderson's when ϵ is small. When ϵ ≪ Θ(1 /m ) , this new bound is superior to both O (1 / ( αϵ )) and ˜ O (1 / ( √ αϵ )) . This superiority is evident when compared to algorithms like ISTA or FISTA [5] to minimize the ℓ 1 -regularization of f for obtaining an approximate solution of Equ. (3). Additionally, when

0

.

0

0

.

5

1

.

0

1

.

5

# Operations

10

6

-

2

.

5

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

ln

‖

π

(

T

)

-

π

‖

1

LocSOR

(

ω

= 1)

LocSOR

(

ω

=

ω

∗

)

APPR

-

12

-

10

-

8

-

6

-

4

/epsilon1

4

6

8

10

12

log(#

Operations

)

Our Upper Bound

Actual Operations

Our Lower Bound

×

ω ∈ (1 , 2) and recalling that Q is an M -matrix, the standard analysis of SOR shows that the spectral norm of the iteration matrix must be larger than | ω -1 | . Hence, 0 < ω < 2 if and only if global SOR converges [55]. When ω ∗ is optimal (the point that the spectral radius of the iteration matrix is minimized), we have the following result.

Corollary 3.4. Let ω = ω ∗ ≜ 2 / (1 + √ 1 -(1 -α ) 2 / (1 + α ) 2 ) and S t = V , ∀ t ≥ 0 during the updates, the global version of LOCSOR has the following convergence bound where ϵ t are small positive numbers with lim t →∞ ϵ t = 0 .

Asymptotically, when ϵ t = o ( √ α ) , then the runtime of global LOCSOR is ˜ O ( m/ √ α ) where ˜ O hides log 1 /ϵ . The main difficulty of analyzing the optimal local LOCSOR is that the nonnegativity and monotonicity of r ( t ) do not hold. Instead, by using a parameterized second-order difference equation, we develop new techniques based on the Chebyshev method detailed in Sec. 4.

3.4 Parallelizable local gradient descent

One disadvantage of LOCSOR is its limited potential for parallelization. The standard GD x ( t +1) = x ( t ) -∇ f ( x ( t ) ) (step size = 1), in contrast, is easy to parallelize across the coordinates of the update. Instead of updating r and x synchronously per-coordinate, we propose the following

Every coordinate in S t is updated in parallel at iteration t . Interestingly, LOCGD exhibits nonnegativity and monotonicity properties, and its runtime complexity is similar to that of LOCSOR, as stated in the following theorem (To remind, ∆ i = 0 for γ T of LOCGD in Equ. (6) ).

Theorem 3.5 (Runtime bound of LOCGD) . Given the configuration θ = ( α, ϵ, s, G ) with α ∈ (0 , 1) and ϵ ≤ 1 /d s and let r ( T ) and x ( T ) be returned by LOCGD defined in (9) for solving Equ. (4) . There exists a real implementation of (9) such that the runtime T LOCGD is bounded by

where C = (1 + α ) / ((1 -α ) |I T | ) , I T = supp( r ( T ) ) . Furthermore, vol( S T ) /γ T ≤ 1 /ϵ and the estimate ˆ π := D 1 / 2 x ( T ) satisfies ∥ D -1 (ˆ π -π ) ∥ ∞ ≤ ϵ .

Note that γ T of LOCGD is empirically smaller than that of LOCSOR. Hence, LOCGD is empirically slower than LOCSOR by only a small constant factor (e.g., twice as slow), a finding consistent with observations of their standard counterparts [19]. Nonetheless, LOCGD is much simpler and more amenable to parallelization on platforms such as GPUs compared to APPR.

4 Accelerated Local Iterative Methods

This section presents our key contributions where we propose faster local methods based on the Chebyshev method for solving Equ. (3) and the Heavy-Ball (HB) method for Equ. (4).

4.1 Local Chebyshev method

Compared with GS and GD, the standard Chebyshev method offers optimal acceleration in solving Equ. (3). Following existing techniques (e.g., see d'Aspremont et al. [11]), we show there exists an upper runtime bound ˜ O ( m/ √ α ) to meeting the stopping condition where ˜ O hides log 1 /ϵ (we presented it in Theorem C.6). Hence, the Chebyshev method is one of the optimal first-order linear solvers for solving Equ. (3). However, localizing Chebyshev poses greater challenges due to the additional momentum vector involved in updating x ( t ) . Our key observation is that if a substantial reduction in the magnitudes of r ( t ) is required within a subset of S t , then the corresponding momentum coordinates are likely to possess significant acceleration energy . Intuitively, a viable strategy involves localizing both the residual and momentum vectors. For t ≥ 1 , denote the 'momentum' vector as ∆ ( t ) := x ( t ) -x ( t -1) and δ t : t +1 = δ t δ t +1 , we propose the localized Chebyshev as the following

where t ≥ 1 with the initials x (0) = 0 , x (1) = r (0) , δ 0 = 0 , δ 1 = (1 -α ) / (1 + α ) , and W = D -1 / 2 AD -1 / 2 is normalized adjacency matrix. Our key strategy for analyzing (10) is to rewrite the updates of r ( t ) as a nonhomogeneous second-order difference equation (see details in Lemma C.8)

where we denote S j,t = S j ∩ · · · ∩ S t -1 ∩ S t given t ≥ j ≥ 0 where S t = V\S t . In the rest, we define ˜ α = (1 - √ α ) / (1 + √ α ) and recall the eigendecomposition of D -1 / 2 AD -1 / 2 = V Λ V ⊤ . Based on the above Equ. (11), we have the following key lemma.

Lemma 4.1. Given t ≥ 1 , x (0) = 0 , x (1) = r (0) . The residual r ( t ) of LOCCH defined in (10) can be expressed as the following

where Z t is a diagonal matrix such that ∥ Z t ∥ 2 ≤ 1 and

where H k,t is a diagonal matrix such that ∥ H k,t ∥ 2 ≤ t -k .

This key lemma essentially captures the process of residual reduction r ( t ) of LOCCH. Specifically, given current iteration t , we define the running residual reduction rate for r ( k ) with k = 0 , 1 , 2 , . . . , t -1 of step t as β k,t , that is,

Note that where whether the last term can be even smaller depends on ∥ r ( k ) S k,j ∥ 2 for S k,j = S k ∩·· ·∩S j -1 ∩S j . However, we notice that the running geometric mean β t ≜ ( ∏ t -1 j =0 (1 + β j )) 1 /t is even smaller in practice. Based on these observations and the assumption on β t , we establish the following theorem.

Theorem 4.2 (Runtime bound of LOCCH) . Given the configuration θ = ( α, ϵ, s, G ) with α ∈ (0 , 1) and ϵ ≤ 1 /d s and let r ( T ) and x ( T ) be returned by LOCCH defined in (10) for solving Equ. (3) . For t ≥ 1 , the residual magnitude ∥ r ( t ) ∥ 2 has the following convergence bound

where y t is a sequence of positive numbers solving y t +1 -2 y t + y t -1 / ((1 + β t -1 )(1 + β t )) = 0 with y 0 = y 1 = ∥ r (0) ∥ 2 . Suppose the geometric mean β t ≜ ( ∏ t -1 j =0 (1 + β j )) 1 /t of β t be such that β t = 1 + c √ α 1 - √ α where c ∈ [0 , 2) . There exists a real implementation of (10) such that the runtime T LOCCH is bounded by

Golub & Overton [18] considered the approximate Chebyshev method by assuming that the inexact residual is sufficiently smaller than ϵ ∥ r ( t ) ∥ 2 , where ϵ must be small enough to ensure convergence. However, this assumption is overly stringent for our case. The novelty of our analysis lies in a more elegant treatment of a parameterized second-order difference equation, allowing us to circumvent this assumption. The nested APGD( ˆ ϵ ), namely ASPR proposed in Martínez-Rubio et al. [37] has runtime complexity ˜ O ( |S ∗ | vol ( S ∗ ) / √ α + |S ∗ | vol ( S ∗ )) where S ∗ is

0

200000

400000

600000

TA

=

vol

(

S

t

)

-

0

.

3

-

0

.

2

-

0

.

1

ln

‖

ˆ

π

-

π

‖

1

ASPR (

ˆ

/epsilon1

=

1.00e-05)

ASPR (

ˆ

/epsilon1

=

1.00e-07)

APPR

0

1

2

TA

=

vol

(

S

t

)

×

10

7

-

0

.

6

-

0

.

4

-

0

.

2

ASPR (

ˆ

/epsilon1

=

3.15e-07)

ASPR (

ˆ

/epsilon1

=

3.15e-09)

APPR

∑

t

∑

t

˜ the optimal support of arg min x { f ( x ) + ϵα ∥ D 1 / 2 x ∥ 1 } and ˜ vol( S ∗ ) = nnz ( Q S ∗ , S ∗ ) . Although it is difficult to compare our bound to this, one limitation of ASPR is that it assumes to call APGD( ˆ ϵ ) O ( |S ∗ | ) times to finish in the worst case. However, our iteration complexity is ˜ O (1 / ( √ α (2 -c ))) . Asymptotically, c = o ( √ α ) ( ϵ → 0 ), our complexity is ˜ O (1 / √ α ) could be better than ˜ O ( |S ∗ | / √ α ) . Fig. 3 presents a preliminary study on ASPR, indicating that it requires more operations than APPR.

We conclude our analysis by presenting a similar result for the local Heavy Ball (HB). Note the HB method is the one when δ t δ t +1 → ˜ α 2 where ˜ α = (1 - √ α ) / (1 + √ α ) . Hence, it has similar convergence analyses as to LOCCH shown in Theorem D.8. The LOCHB has the following updates

5 Generalization and Open Problems

Our framework can be applied to various local methods for large-scale linear systems. Extensions of this framework to other linear systems are detailed in Tab. 2 of Appendix E. More broadly, we consider the feasibility of local methods for solving Qx = b , where b is a sparse vector ( | supp( b ) |≪ n ) and Q is a positive definite, graph-induced matrix with bounded eigenvalues. This leads us to question whether all standard iterative methods can be effectively localized, raising two key questions

Given a graph-induced matrix Q and its spectral radius ρ ( Q ) < 1 , a standard solver A , and the corresponding local evolving process Φ θ ( S t , x ( t ) , r ( t ) , LOC A ) , does a localized version of A (over S t ) converge and have local runtime bounds?

Based on current analysis, Theorem 4.2 relies on the geometric mean of residual reduction on ∥ r ( k ) ∥ 2 being small. How feasible is acceleration within locality constraints? Specifically, a stronger bound could be established for solving Equ. (3) via LOCHB and LOCCH, with a graph-independent bound of

Additionally, this work primarily focuses on using first-order neighbors at each iteration. An area for future exploration is generalizing to higher-order neighbors to determine if this leads to faster or more efficient methodologies, which remains an open question.

6 Experiments

We conduct experiments over 17 graphs to solve (3) and explore the local clustering task. We address the following questions: 1) Can iterative solvers be effectively localized? 2) How does the performance of accelerated local methods compare to non-accelerated ones? 3) Can our proposed methods reduce the number of operations required for local clustering? 6

Baselines. We consider four baselines: 1) Conjugate Gradient Method (CGM) as a benchmark to compare local and non-local methods; 2) ISTA, the local method proposed by Fountoulakis et al. [13]; 3) FISTA, the momentum-based local algorithm proposed by Hu [22]; and 4) APPR, the classic local method proposed by Andersen et al. [2]. All methods are implemented in Python 3.10 with the numba library [33].

Efficiency of localized algorithms. To compare local solvers to their standard counterparts, we set α = 0 . 1 , randomly select 50 nodes from each graph to serve as e s in (3), and run standard GD, SOR, HB, and CH solvers along with their local counterparts: LOCGD, LOCSOR, LOCHB, and LOCCH. We measure the efficiency by the speedup , defined as the ratio between the runtime of the standard and local solver. The range of ϵ is ϵ ∈ [ α 2(1+ α ) d s , 10 -4 /n ] . The results, presented in Fig. 4, clearly indicate that our design demonstrates significant speedup, especially around ϵ = 1 /n . Remarkably, they still show better performance even when ϵ ≈ 10 -4 /n (Fig. 5). These results suggest that local solvers are preferred over non-local ones when the precision requirement is in this range.

10

-

8

10

-

5

10

-

2

0

50

100

150

Speedup

ogbl-ppa

/

LOCGD

SOR

/

LOCSOR

HB

/

LOCHB

CH

/

LOCCH

10

-

8

10

-

5

10

-

2

0

20

40

60

ogbn-arxiv

10

-

8

10

-

5

10

-

2

0

20

40

60

80

ogbn-mag

10

-

8

10

-

5

10

-

2

0

100

200

300

ogbn-products

10

-

8

10

-

5

10

-

2

0

50

100

ogbn-proteins

/epsilon1

/epsilon1

/epsilon1

/epsilon1

/epsilon1

Comparison with local baselines and CGM. We next compare our three accelerated methods with four baselines. Fig. 5 presents the ℓ 1 -estimation error in terms of the number of operations (quantified as t · vol( S t ) ) executed. It is evident that our three solvers use significantly fewer operations compared to CGM and the other three local methods. Again, due to maintaining a nondecreasing set of active nodes, ISTA and FISTA require more operations

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

Operations

10

9

-

6

-

4

-

2

ln

‖

x

(

T

)

-

x

∗

‖

1

ogbl-ppa

LocSOR

LocHB

LocCH

ISTA

FISTA

CGM

APPR

0

1

2

3

Operations

10

9

-

8

-

6

-

4

-

2

ogbn-products

×

×

than the locally evolving set process. Ours are more efficient than APPR, where r (0) = e s is used.

0

1

2

Operations

10

7

-

3

.

5

-

3

.

0

-

2

.

5

-

2

.

0

ln

‖

x

(

T

)

-

x

‖

1

com-friendster

APPR

LocHB

LocCH

LocSOR

0

2

4

Operations

10

7

-

3

-

2

ogbn-papers100M

×

×

Efficiency in terms of α and huge-graph tests. We demonstrate the performance of local solvers in terms of different α ranging from 0 . 005 to 0 . 25 . Interestingly, in Fig. 13, LOCGD show faster convergence when α is small; this may be because of the advantages of monotonicity properties, which is not present in the accelerated methods. However, in other regions of α , accelerated methods are faster. We also tested local solvers on two large-scale graphs where pa- pers100M has 111M nodes and 1.6B edges while com-friendster has 65M nodes with 1.8B edges. Results are shown in Fig. 6; compared with current default local methods, it is several times faster, especially on ogbn-papers100M.

Case study on local clustering. Following the experimental setup in Fountoulakis et al. [13], we consider the task of local clustering on 15 graphs. As partially demonstrated in Tab. 1, compared with APPR and FISTA, LOCSOR uses the least operations and is the fastest, demonstrating the advantages of our proposed local solvers.

7 Limitations and Conclusion

Our proposed algorithms may have the following limitations: 1) When α is small, the acceleration effect partially disappears, as observed in Fig. 13. This may be due to the limitations of global counterparts, where the residual may not decrease early; 2) Our new accelerated bound for LocCH depends on an empirically reasonable assumption of residual reduction but lacks theoretical justification.

We propose using a new locally evolving set process framework to characterize algorithm locality and demonstrate that several standard iterative solvers can be effectively localized, significantly speeding up current local solvers. Our local methods could be efficiently implemented into GPU architecture to accelerate the training of GNNs such as APPNP [27] and PPRGo [7]. We also offer open problems in developing faster local methods. It is worth exploring whether subsampling active nodes stochastically or using different queue strategies (priority rather than FIFO) could help speed up the framework further. It also remains interesting to see how to design local algorithms for conjugate direction-based methods such as CGM.

Acknowledgments and Disclosure of Funding

The authors would like to thank the anonymous reviewers for their helpful comments. The work of Baojian Zhou is sponsored by Shanghai Pujiang Program (No. 22PJ1401300) and the National Natural Science Foundation of China (No. KRH2305047). The work of Deqing Yang is supported by Chinese NSF Major Research Plan No.92270121. The computations in this research were performed using the CFFF platform of Fudan University.

References

Alon, N., Rubinfeld, R., Vardi, S., and Xie, N. Space-efficient local computation algorithms. In Proceedings of the twenty-third annual ACM-SIAM symposium on Discrete Algorithms (SODA) , pp. 1132-1139. SIAM, 2012.

Andersen, R., Chung, F., and Lang, K. Local graph partitioning using PageRank vectors. In 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS) , 2006.

Andersen, R., Gharan, S. O., Peres, Y., and Trevisan, L. Almost optimal local graph clustering using evolving sets. Journal of the ACM (JACM) , 63(2):1-31, 2016.

Anikin, A., Gasnikov, A., Gornov, A., Kamzolov, D., Maximov, Y., and Nesterov, Y. Efficient numerical methods to solve sparse linear equations with application to PageRank. Optimization Methods and Software , pp. 1-29, 2020.

Beck, A. and Teboulle, M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. , 2:183-202, 2009.

Berkhin, P. Bookmark-coloring algorithm for personalized pagerank computing. Internet Mathematics , 3(1):41-62, 2006.

Bojchevski, A., Klicpera, J., Perozzi, B., Kapoor, A., Blais, M., Rózemberczki, B., Lukasik, M., and Günnemann, S. Scaling graph neural networks with approximate pagerank. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , 2020.

Boyd, S., Diaconis, P., and Xiao, L. Fastest mixing markov chain on a graph. SIAM review , 46 (4):667-689, 2004.

Chen, Z., Guo, X., Zhou, B., Yang, D., and Skiena, S. Accelerating personalized PageRank vector computation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) , pp. 262-273, 2023.

Chung, F. The heat kernel as the pagerank of a graph. Proceedings of the National Academy of Sciences (PNAS) , 104(50):19735-19740, 2007.

d'Aspremont, A., Scieur, D., Taylor, A., et al. Acceleration methods. Foundations and Trends® in Optimization , 5(1-2):1-245, 2021.

Fountoulakis, K. and Yang, S. Open problem: Running time complexity of accelerated ℓ 1 -regularized PageRank. In Conference on Learning Theory (COLT) , 2022.

Fountoulakis, K., Roosta-Khorasani, F., Shun, J., Cheng, X., and Mahoney, M. W. Variational perspective on local graph clustering. Mathematical Programming , 174(1):553-573, 2019.

Fountoulakis, K., Wang, D., and Yang, S. p-norm flow diffusion for local graph clustering. In ICML , 2020.

Gasteiger, J., Weißenberger, S., and Günnemann, S. Diffusion improves graph learning. In Advances in neural information processing systems (NeurIPS) , 2019.

Gleich, D. and Mahoney, M. Anti-differentiating approximation algorithms: A case study with min-cuts, spectral, and flow. In International Conference on Machine Learning , pp. 1018-1025. PMLR, 2014.

Gleich, D. F. PageRank beyond the web. siam REVIEW , 57(3):321-363, 2015.

Golub, G. H. and Overton, M. L. The convergence of inexact chebyshev and richardson iterative methods for solving linear systems. Numerische Mathematik , 53(5):571-593, 1988.

Golub, G. H. and Van Loan, C. F. Matrix computations (4th Edition) . JHU press, 2013.

Guo, X., Zhou, B., and Skiena, S. Subset node representation learning over large dynamic graphs. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pp. 516-526, 2021.

Guo, X., Zhou, B., and Skiena, S. Subset node anomaly tracking over large dynamic graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pp. 475-485, 2022.

Hu, C. Local graph clustering using l1-regularized PageRank algorithms. Master's thesis, University of Waterloo, 2020.

Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Johnson, R. and Zhang, T. Graph-based semi-supervised learning and spectral kernel design. IEEE Transactions on Information Theory , 54(1):275-288, 2008.

Kapralov, M., Lattanzi, S., Nouri, N., and Tardos, J. Efficient and local parallel random walks. Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Klamkin, M. and Newman, D. J. Extensions of the weierstrass product inequalities. Mathematics Magazine , 43(3):137-141, 1970.

Klicpera, J., Bojchevski, A., and Günnemann, S. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations (ICLR) , 2019.

Kloster, K. and Gleich, D. F. A nearly-sublinear method for approximating a column of the matrix exponential for matrices from large, sparse networks. In Algorithms and Models for the Web Graph: 10th International Workshop, WAW 2013, Cambridge, MA, USA, December 14-15, 2013, Proceedings 10 , pp. 68-79. Springer, 2013.

Kloster, K. and Gleich, D. F. Heat kernel based community detection. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD) , 2014.

Kloumann, I. M. and Kleinberg, J. M. Community membership identification from small seed sets. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD) , 2014.

Koutis, I., Miller, G. L., and Peng, R. A nearly-m log n time solver for sdd linear systems. In 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science (FOCS) , pp. 590-598. IEEE, 2011.

Ł ˛ acki, J., Mitrovi´ c, S., Onak, K., and Sankowski, P. Walking randomly, massively, and efficiently. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing (STOC) , pp. 364-377, 2020.

Lam, S. K., Pitrou, A., and Seibert, S. Numba: A llvm-based python jit compiler. In Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC , pp. 1-6, 2015.

Leskovec, J., Lang, K. J., Dasgupta, A., and Mahoney, M. W. Community structure in large networks: Natural cluster sizes and the absence of large well-defined clusters. Internet Mathematics , 6(1):29-123, 2009.

Leventhal, D. and Lewis, A. S. Randomized methods for linear constraints: convergence rates and conditioning. Mathematics of Operations Research , 35(3):641-654, 2010.

Mahoney, M. W., Orecchia, L., and Vishnoi, N. K. A local spectral method for graphs: With applications to improving graph partitions and exploring data graphs locally. The Journal of Machine Learning Research (JMLR) , 13(1):2339-2365, 2012.

Martínez-Rubio, D., Wirth, E., and Pokutta, S. Accelerated and sparse algorithms for approximate personalized PageRank and beyond. In Proceedings of Thirty Sixth Conference on Learning Theory (COLT) , volume 195 of Proceedings of Machine Learning Research , pp. 2852-2876. PMLR, 2023.

Morris, B. and Peres, Y. Evolving sets and mixing. In Proceedings of the Thirty-Fifth Annual ACM Symposium on Theory of Computing (STOC) , pp. 279-286, New York, NY, USA, 2003. Association for Computing Machinery.

Nutini, J., Schmidt, M., Laradji, I., Friedlander, M., and Koepke, H. Coordinate descent converges faster with the gauss-southwell rule than random selection. In ICML , pp. 1632-1641. PMLR, 2015.

Polyak, B. T. Introduction to optimization . New York, Optimization Software' 1987.

Rakhlin, A. and Sridharan, K. Efficient online multiclass prediction on graphs via surrogate losses. In Artificial Intelligence and Statistics , pp. 1403-1411. PMLR, 2017.

Rubinfeld, R. and Shapira, A. Sublinear time algorithms. SIAM Journal on Discrete Mathematics , 25(4):1562-1588, 2011.

Saad, Y. Iterative methods for sparse linear systems . SIAM, 2003.

Schaub, M. T., Benson, A. R., Horn, P., Lippner, G., and Jadbabaie, A. Random walks on simplicial complexes and the normalized hodge 1-laplacian. SIAM Review , 62(2):353-391, 2020.

Spielman, D. A. Algorithms, graph theory, and linear equations in laplacian matrices. In Proceedings of the International Congress of Mathematicians 2010 (ICM 2010) (In 4 Volumes) Vol. I: Plenary Lectures and Ceremonies Vols. II-IV: Invited Lectures , pp. 2698-2722. World Scientific, 2010.

Spielman, D. A. and Teng, S.-H. Nearly linear time algorithms for preconditioning and solving symmetric, diagonally dominant linear systems. SIAM Journal on Matrix Analysis and Applications , 35(3):835-885, 2014.

Stevi´ c, S. Bounded solutions to nonhomogeneous linear second-order difference equations. Symmetry , 9(10):227, 2017.

Teng, S.-H. et al. Scalable algorithms for data and network analysis. Foundations and Trends® in Theoretical Computer Science , 12(1-2):1-274, 2016.

Tseng, P. and Yun, S. A coordinate gradient descent method for nonsmooth separable minimization. Mathematical Programming , 117:387-423, 2009.

Tu, S., Venkataraman, S., Wilson, A. C., Gittens, A., Jordan, M. I., and Recht, B. Breaking locality accelerates block gauss-seidel. In ICML , pp. 3482-3491. PMLR, 2017.

Vishnoi, N. K. et al. Lx= b. Foundations and Trends in Theoretical Computer Science , 8(1-2): 1-141, 2013.

Wang, J.-K., Lin, C.-H., and Abernethy, J. D. A modular analysis of provable acceleration via Polyak's momentum: Training a wide ReLU network and a deep linear network. In ICML , pp. 10816-10827. PMLR, 2021.

Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., and Weinberger, K. Simplifying graph convolutional networks. In ICML , pp. 6861-6871. PMLR, 2019.

Yin, H., Benson, A. R., Leskovec, J., and Gleich, D. F. Local higher-order graph clustering. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining (KDD) , 2017.

Young, D. M. Iterative solution of large linear systems . Elsevier, 2014.

Zeng, H., Zhang, M., Xia, Y., Srivastava, A., Malevich, A., Kannan, R., Prasanna, V., Jin, L., and Chen, R. Decoupling the depth and scope of graph neural networks. Advances in Neural Information Processing Systems , 34:19665-19679, 2021.

Zhou, B., Sun, Y., and Harikandeh, R. B. Fast online node labeling for very large graphs. In ICML , pp. 42658-42697. PMLR, 2023.

Appendix / supplemental material

A Notations and Proof of Lemma 2.1

A.1 List of Notations

In the rest of the appendix, we use the following notations:

A.2 Proof of Lemma 2.1

Lemma 2.1 (Runtime bound of APPR [2]) . Given α ∈ (0 , 1) and the precision ϵ ≤ 1 /d s for node s ∈ V with p ← 0 , z ← e s at the initial, APPR ( G , ϵ, α, s ) defined in (2) returns an estimate p of π . There exists a real implementation of (2) (e.g., Algo. 2) such that the runtime T APPR satisfies

Furthermore, the estimate ˆ π := p satisfies ∥ D -1 (ˆ π -π ) ∥ ∞ ≤ ϵ and vol(supp(ˆ π )) ≤ 2 / ((1 -α ) ϵ ) .

Proof. To find an upper bound of T APPR, we add a time index for all active nodes u 1 , u 2 , . . . , u t processed in APPR of Algo. 2, a real implementation of (2). The parameter t is the number of active nodes processed, and u i is the node dequeued at time i . So, updates of p and z are from ( 0 , e s ) = ( p (0) , z (0) ) to ( p ( t ) , z ( t ) ) as follows:

For each active u i , the updates of ( p , z ) by definition can be represented as

Since z (0) = e s ≥ 0 , then z ( i ) ≥ 0 and p ( i ) ≥ 0 for all i by induction. Note that ∥ AD -1 e u i ∥ 1 = 1 , we have the following relation from the updates of z

Note ϵd u i ≤ z ( i -1) u i for each active u i . Summing the above equation over i = 1 , 2 , . . . , t , we have where note ∥ z (0) ∥ 1 = ∥ e s ∥ 1 = 1 by the initial condition. Since ∑ t i =1 d u i exactly captures the number of operations needed, the runtime of Algo. 2 is then bounded as

To check the quality of estimate p , using the updates of p ( i ) and summing over all i , we have

where Π is the PPR matrix. The above gives us π -p ( t ) = Π · z ( t ) . Since G is undirected, the Π matrix satisfies π v [ u ] = ( d u /d v ) π u [ v ] where π v [ u ] is the u -th element of PPR vector of sourcing node v . Consider each u -th element of Π · z ( t )

where the last equality is due to ∑ v ∈V π u [ v ] = 1 . Hence, ( π -p ( t ) ) u = ( Π · z ( t ) ) u ≤ ϵd u , which indicates ∥ D -1 ( p ( t ) -π ) ∥ ∞ ≤ ϵ . To see the bound of vol(supp( p ( t ) )) , note for any u ∈ supp( p ( t ) ) , it was an active node and there was at least ˜ z u (1 -α ) / 2 remain in u -th entry of z ( t ) where we denote ˜ z u as the residual before the last push operation of node u ; hence

Algo. 2 APPR ( α, ϵ, s, G ) via FIFO Queue

Indeed, Lemma 2.1 is a special case of Theorem 1 in [2]. The proof outlined above adheres to the key strategy demonstrated in that theorem, which involves exploring the monotonicity and nonnegativity of z .

The real implementation of APPR, as shown in Algo. 2, presents a typical queue-based method. It has monotonic properties during the updates of p and z (Lines 10-16 of Algo. 2). It also holds element-wise that p ≥ 0 and z ≥ 0 . The operations of Q .enqueue( u ), Q .dequeue(), and v / ∈ Q are all in O (1) . Line 4 to Line 9 is to design the marker for distinguishing between S t and S t +1 . If all active nodes are processed and no more active nodes are added into Q , then Q will be empty, and finally, the algorithm returns an estimate p of π .

B Local Iterative Methods via Evolving Set Process

Justification of an equivalent condition. We make the justification of an equivalent stop condition for solving (3). Note we require a local solver to return an estimate ˆ π satisfies

Since we define Qx = b as ( I -1 -α 1+ α D -1 / 2 AD -1 / 2 ) x = 2 α 1+ α D -1 / 2 e s . With r ( t ) = b -Qx ( t ) and the stop condition ensures the estimate ˆ π = D 1 / 2 x ( t ) satisfies (14). To see this, since π = D 1 / 2 x ∗ , we have

where D -1 / 2 Q -1 D 1 / 2 = ( I -1 -α 1+ α D -1 A ) -1 = ∑ ∞ i =0 ( 1 -α 1+ α D -1 A ) i . This leads to

B.1 Local Variant of GS-SOR and Proof of Lemma 3.2

The Gauss-Seidel Successive Over-Relaxation (GS-SOR) solver (see Section 11.2.7 of Golub & Van Loan [19]) for the linear system Mπ = s via the following forward substitution for i in V := { 1 , 2 , . . . , n } do :

where p is updated from p ( t ) to p ( t +1) . When the relaxation parameter ω = 1 , GS-SOR reduces to the standard GS method. Equivalently, let ∆ i = ( i -1) /n for i = 1 , 2 , . . . , n , then GS-SOR updates can be sequentially represented as

for i in V := { 1 , 2 , . . . , n } do :

Therefore, it is natural to define the following local variant of GS-SOR.

Definition B.1 (Local variant of GS-SOR) . Consider the linear system Mπ = s . For t ≥ 0 , we are given an active node set S t = { u 1 , u 2 , . . . , u |S t | } and let ∆ i = ( i -1) / |S t | for i = 1 , 2 , . . . , |S t | , providing ω ∈ (0 , 2) , it is natural to define the local variant of GS-SOR as follows:

for u i in S t := { u 1 , u 2 , . . . , u |S t | } do :

where S t ⊆ V . When ω = 1 and S t = V = { 1 , 2 , . . . , n } , it reduces to the standard GS.

We use the above definition to show APPR is a local variant of GS-SOR as the following. Lemma 3.2 (New local evolving-based bound for APPR) . Let M = α -1 ( I -1 -α 2 ( I + AD -1 ) ) and s = e s . The linear system Mπ = s is equivalent to Equ. (1) . Given p (0) = 0 , z (0) = e s with ω ∈ (0 , 2) , the local variant of GS-SOR (15) for Mπ = s can be formulated as

where u i is an active node in S t satisfying z u i ≥ ϵd u i and ∆ i = ( i -1) / |S t | . Furthermore, when ω = 1+ α 2 , this method reduces to APPR given in (7) , and there exists a real implementation (Aglo. 2) of APPR such that the runtime T APPR is bounded by

Proof. Note that Mπ = s is equivalent to Equ. (1). We first rewrite p ( t +∆ i +1 ) in terms of the residual z . The residual z at time t +∆ i can be written as z ( t +∆ i ) = s -Mp ( t +∆ i ) . Note s u i -∑ i -1 j =1 M u i u j p ( t +∆ i ) u j -∑ n j = i M u i u j p ( t +∆ i ) u j = ( s -Mp ( t +∆ i ) ) u i = z ( t +∆ i ) u i . Then, the updates of local GS-SOR defined in (15) can be rewritten as p ( t +∆ i +1 ) = p ( t +∆ i ) + ω M u i u i z ( t +∆ i ) u i · e u i . Hence, the updates of z ( t +∆ i ) can be written as

Note the diagonal element M u i u i = (1 + α ) / (2 α ) . Hence, when ω = 1+ α 2 , we have

The above updates match APPR's evolving set process formulation in (7). The rest is to show a new runtime bound. Adding ℓ 1 -norm on both sides of the above equation, then note ∥ z ( t +∆ i +1 ) ∥ 1 = ∥ z ( t +∆ i ) ∥ 1 -αz ( t +∆ i ) u i for i = 1 , 2 , . . . , |S t | . We have where we define β t := ∑ |S t | i =1 | z ( t +∆ i ) u i | ∥ z ( t ) ∥ 1 . Let t = T -1 , we have

̸

where the first inequality is due to ln(1 + x ) ≤ x for x > -1 . For each nonzero node u ∈ I T = { z ( T ) u : z ( T ) u = 0 , u ∈ V} , consider the last time t ′ that it was altered. Then, either the alteration came from u being an active node, with z ( t ′ ) u ≥ d u ϵ , and after the PUSH operation it became z ( t ′′ ) u ≥ (1 -α ) d u 2 ϵ ; or the alteration came from a neighboring node v u ∈ N ( u ) pushing its mass onto u , which ensures that z ( t ′′ ) u ≥ (1 -α ) 2 d vu z ( t ′ ) v u ≥ (1 -α ) 2 ϵ . These two cases provide a lower bound of 1 -α 2 ϵ . Hence, ∥ z ( T ) ∥ 1 ≥ ϵ (1 -α ) |I T | 2 , which leads to the corresponding constant C T .

To see the lower bound of 1 /ϵ , note ϵd u i ≤ z ( t +∆ i ) u i for all i = 1 , 2 , . . . , |S t | . Then we have

where we defined β t := ∑ |S t | i =1 | z ( t +∆ i ) u i | ∥ z ( t ) ∥ 1 and the last inequality is due to the monotonic decreasing of ∥ z ( t ) ∥ 1 , i.e., 1 ≥ ∥ z (0) ∥ 1 ≥ · · · ≥ ∥ z ( T ) ∥ 1 . Applying the above inequality for all t = 0 , 1 , 2 . . . , T -1 , it leads to

where the last derivation is from the fact that ˆ γ T = 1 T { ∑ T -1 t =0 β t := ∑ |S t | i =1 | z ( t +∆ i ) u i | ∥ z ( t ) ∥ 1 } .

Remark B.2 . The connection between APPR and the Gauss-Seidel is not new [28, 29, 16, 9]. Our work is the first work that has linked APPR and the Gauss-Seidel with a locally evolving set process.

B.2 LOCSOR and Proof of Theorem 3.3

In this subsection, recall we defined ˜ r ( t ) = D 1 / 2 r ( t ) .

Lemma B.3 (Local iteration complexity of LOCSOR ( ω ≤ 1 )) . Denote S t = { u 1 , u 2 , . . . , u |S t | } as the active node set at the t -th iteration. When ω ∈ (0 , 1] , all vectors ˜ r ( t ) ≥ 0 are nonnegative and magnitudes are decreasing ∥ ˜ r ( t +1) ∥ 1 < ∥ ˜ r ( t ) ∥ 1 . Let T be the total number of iterations needed. Then, at iteration T , we have where γ t = t -1 ∑ t -1 τ =0 γ τ is the mean of active ratio factors defined in Equ. (6) . Proof. Recall u i ∈ S t = { u 1 , . . . , u |S t | } and ∆ i = i -1 |S t | , LOCSOR in Algo. 3 updates x ( t +∆ i +1 ) = x ( t +∆ i ) + ωr ( t +∆ i ) u i · e u i r ( t +∆ i +1 ) = r ( t +∆ i ) -ωr ( t +∆ i ) u i · e u i + (1 -α ) ω 1+ α r ( t +∆ i ) u i · D -1 / 2 AD -1 / 2 e u i Note r ≥ 0 during updates when ω ∈ (0 , 1] and recall ˜ r ( t ) = D 1 / 2 r ( t ) , we have ∥ ˜ r ( t +∆ i +1 ) + ω ˜ r ( t +∆ i ) u i · e u ∥ 1 = ∥ ˜ r ( t +∆ i ) + (1 -α ) ω 1+ α ˜ r ( t +∆ i ) u i · AD -1 e u i ∥ 1 ∥ ˜ r ( t +∆ i +1 ) ∥ 1 + ω ˜ r ( t +∆ i ) u i = ∥ ˜ r ( t +∆ i ) ∥ 1 + (1 -α ) ω 1+ α ˜ r ( t +∆ i ) u i .

Summing over the above equations over u i , we have

Given { x i } T -1 i =0 and x i ∈ (0 , 1) , the Weierstrass product inequality provides 1 -∑ T -1 i =0 x i ≤ ∏ T -1 i =0 (1 -x i ) . By using this inequality, we continue to have a lower bound of T as the following

∥ ∥ To get upper bound of γ t , note each active residual ˜ r ( t +∆ i ) u i pushes at most (1 -α ) ω (1+ α ) times magnitude to ˜ r u i +1 , ˜ r u i +2 , and ˜ r u |S t | ; hence, ∑ |S t | j = i ˜ r ( t +∆ j ) u j will increase by at most ˜ r ( t +∆ i ) u i · (1 -α ) ω (1+ α ) ≤ ˜ r ( t +∆ i ) u i in total. Hence, overall u i , we have

We reach the following lower and upper bounds of γ t , ∥ ˜ r ( t ) S t ∥ 1 ∥ ˜ r ( t ) ∥ 1 ≤ γ t := ∑ |S t | i =1 ˜ r ( t +∆ i ) u i ∥ ˜ r ( t ) ∥ 1 ≤ 2 ∥ ˜ r ( t ) S t ∥ 1 ∥ ˜ r ( t ) ∥ 1 . To check the upper bound of T , from Equ. (17), ∥ ˜ r ( T ) ∥ 1 = ∏ T -1 t =0 ( 1 -2 αωγ t 1+ α ) ∥ ˜ r (0) ∥ 1 and where the first inequality is due to ln(1 + x ) ≤ x for x > -1 .

Theorem 3.3 (Runtime bound of LOCSOR ( ω = 1) ) . Given the configuration θ = ( α, ϵ, s, G ) with α ∈ (0 , 1) and ϵ ≤ 1 /d s and let r ( T ) and x ( T ) be returned by LOCSOR defined in (8) for solving Equ. (3) . There exists a real implementation of (8) such that the runtime T LOCSOR is bounded by where vol( S T ) and γ T are defined in (6) and C = 1+ α (1 -α ) |I T | with I T = supp( r ( T ) ) . Furthermore, vol( S T ) /γ T ≤ 1 /ϵ and the local estimate ˆ π := D 1 / 2 x ( T ) satisfies ∥ D -1 (ˆ π -π ) ∥ ∞ ≤ ϵ .

̸

Proof. After the last iteration T , for each nonzero residual ˜ r ( T ) u = 0 , u ∈ I T , there must be at least one update that happened at node u : Node u has a neighbor v u ∈ N ( u ) , which was active. This neighbor v u pushed some residual (1 -α )˜ r ( t ′ ) vu (1+ α ) d vu to u where t ′ < T . Hence, for all u ∈ I T , we have where the second inequality is because ˜ r ( t ′ ) v u was active before the push operation. Applying the above lower bound of ∥ ˜ r ( T ) ∥ 1 to Equ. (16) of Lemma B.3 and note ∥ ˜ r (0) ∥ 1 = 2 α/ (1 + α ) , we obtain

The rest is to prove an upper bound 1 / ( αϵ ) of T LOCSOR. Recall that for any active node u , we have residual updates from Algo. 3 as the following

Move -ωr ( t ) u D 1 / 2 e u to the left and note ∥ AD -1 D 1 / 2 e u ∥ 1 = √ d u , we then obtain

Hence, for each active u , we have 2 αω √ d u r ( t ) u 1+ α = ∥ D 1 / 2 r ( t ) ∥ 1 -∥ D 1 / 2 r ( t +1) ∥ 1 . Summing them over all active nodes u and noticing r ( t ) u ≥ 2 αϵ √ d u / (1 + α ) by the active condition. Note ω = 1 and || D 1 / 2 r (0) || 1 = 2 α 1+ α , we have run time bounded by

Combining the above bound and the bound T shown in Lemma B.3, we prove the lower and upper bound of T LocSOR . To check the lower bound of 1 /ϵ ,i.e., vol( S T ) /γ T ≤ 1 /ϵ , note 2 αϵd u i 1+ α ≤ ˜ r ( t +∆ i ) u i for all i = 1 , 2 , . . . , |S t | . Then we have

where the last inequality is due to the monotonic decreasing of ∥ D 1 / 2 r ( t ) ∥ 1 , i.e., 2 α 1+ α ≥ ∥ D 1 / 2 r (0) ∥ 1 ≥ ·· · ≥ ∥ r ( T ) ∥ 1 . Applying the above inequality over all t = 0 , 1 , 2 . . . , T -1 , it leads to

The real queue-based implementation of LOCSOR is presented in Algo. 3. It has monotonic and nonnegative properties during the updates of r ≥ 0 and x ≥ 0 when ω ∈ (0 , 1] . Same as APPR, the operations of Q .enqueue( u ), Q .dequeue(), and v / ∈ Q are all in O (1) .

During the updates, one should note that the real vector r presents D 1 / 2 r ( t ) while the vector x is D 1 / 2 x ( t ) . In this case, the original active node condition is implicitly shifting from | r u | ≥ 2 αϵ √ d u 1+ α to √ d u | r u | ≥ 2 αϵd u 1+ α . Weuse this shifted active condition in Lines 11 and 13 and inactive condition in Line 5. When ω ∈ (1 , 2) , it is possible | r u | < c · ϵd u and LOCSOR will ignore this inactive node u during the updates. This step makes sure S t = { u i : | r ( t +∆ i ) u i | ≥ 2 αϵ √ d u 1+ α } during the updates.

B.3 Optimal GS-SOR and Proof of Corollary 3.4

We introduce the following standard result.

Lemma B.4 (Young [55], Section 12.2, Theorem 2.1 ) . Given the GS-SOR method for solving Qx = b , if the underlying matrix Q is a Stieltjes matrix and set relaxation parameter ω as

where ρ ( B ) is the largest eigenvalue (in magnitude) of B = I -diag ( Q ) -1 Q , then where

L

ω

diag

:= (

(

Q

)

ω

Q

L

)

-

1

Corollary 3.4. Let ω = ω ∗ ≜ 2 / (1 + √ 1 -(1 -α ) 2 / (1 + α ) 2 ) and S t = V , ∀ t ≥ 0 , the global version of LOCSOR has the following convergence bound

(

ω

Q

U

-

(

ω

-

1)

diag

(

Q

))

with

Q

=

diag

(

Q

)

-

Q

U

-

Q

L

.

where ϵ t are small positive numbers with lim t →∞ ϵ t = 0 .

-

Proof. Recall Q = I -1 -α 1+ α D -1 / 2 AD -1 / 2 and we consider the underlying graph as simple which means A has 0 diagonal. Hence, diag ( Q ) = I and B is defined as

Since Q is a Stieltjes matrix, then Lemma B.4 gives a bound on the spectral radius of L ω as

Recall that Gelfand's formula states [52]: Given spectral radius ρ ( L ω ∗ ) := max i ∈ [ n ] | λ i ( L ω ∗ ) | , where λ i ( · ) is the i -th eigenvalue, there exists a sequence { ϵ t } ≥ 0 such that ∥ L t ω ∗ ∥ 2 = ( ρ ( L ω ∗ ) + ϵ t ) t and lim t →∞ ϵ t = 0 . The standard SOR method is defined as the following

Note r ( t ) = Qe ( t ) = QL t ω ∗ Q -1 Qe (0) = QL t ω ∗ Q -1 r (0) . We have

To meet the stop condition, we require | r ( t ) u | ≤ 2 αϵ √ d u 1+ α . It is enough to make sure ∥ r ( t ) ∥ 2 ≤ 2 αϵ (1+ α ) √ d s . This leads to find t such that

When ϵ t = o ( √ α ) , then the runtime of global LOCSOR is ˜ O ( m/ √ α ) where ˜ O hides log 1 ϵ .

B.4 LOCGD and Proof of Theorem 3.5

The local gradient descent, namely LOCGD is to use x ( t +1) = x ( t ) + r ( t ) S t and r ( t +1) = r ( t ) -Qr ( t ) S t , where S t = { u i : | r ( t +∆ i ) u i | ≥ 2 αϵ √ d u / (1 + α ) } where ∆ i = 0 . Algo. 4 presents our actual implementation of LOCGD via FIFO Queue.

Algo. 4 presents LOCGD similar to the real queue-based implementation of LOCSOR. It has monotonic and nonnegative properties during the updates of r ≥ 0 and x ≥ 0 . Again, the operations of Q .enqueue( u ), Q .dequeue(), and v / ∈ Q are all in O (1) . During the updates, one should note that r presents D 1 / 2 r ( t ) while x is D 1 / 2 x ( t ) . All shifted conditions are the same as of LOCSOR. The key advantage of LOCGD is that it is highly parallelizable, while LOCSOR is truly an online update, so it is hard to parallelize.

Lemma B.5 (Iterations of LOCGD) . With the initial x (0) = 0 , r (0) = b , S 0 = supp( r (0) ) , denote ˜ r ( t ) = D 1 / 2 r ( t ) . LOCGD defined in (9) has the following properties: 1) x ( t ) ≥ 0 , r ( t ) ≥ 0 and ∥ r ( t ) ∥ ≥ ∥ r ( t +1) ∥ 1 ; 2) The residual and estimation error satisfies

Proof. We first show x ( t ) ≥ 0 , r ( t ) ≥ 0 are all nonnegative vectors during the updates when b ≥ 0 . This can be seen from the induction. At the initial stage, x (0) ≥ 0 and r (0) = b ≥ 0 . Now assume that for any t ≥ 0 , x ( t ) ≥ 0 and r ( t ) ≥ 0 . Then x ( t +1) = x ( t ) + r ( t ) S k ≥ 0 , and r ( t +1) = r ( t ) S t + 1 -α 1+ α D -1 / 2 AD -1 / 2 r ( t ) S t ≥ 0 . Therefore, x ( t ) ≥ 0 and r ( t ) ≥ 0 for all t . Note ˜ r ( t +1) = ˜ r ( t ) S t + 1 -α 1+ α AD -1 · ˜ r ( t ) S t and since ∥ AD -1 ˜ r ( t ) S t ∥ 1 = ∥ ˜ r ( t ) S t ∥ 1 , we will have

Then, we can bound the total residual as the following theorem.

Theorem 3.5 (Runtime bound of LOCGD) . Given the configuration θ = ( α, ϵ, s, G ) with α ∈ (0 , 1) and ϵ ≤ 1 /d s and let r ( T ) and x ( T ) be returned by LOCGD defined in (9) for solving Equ. (4) . There exists a real implementation of (9) such that the runtime T LOCGD is bounded by

where C = (1 + α ) / ((1 -α ) |I T | ) , I T = supp( r ( T ) ) . Furthermore, vol( S T ) /γ T ≤ 1 /ϵ and the estimate ˆ π := D 1 / 2 x ( T ) satisfies ∥ D -1 (ˆ π -π ) ∥ ∞ ≤ ϵ .

Proof. We first show bound 1 / ( αϵ ) . We first rearrange r ( t +1) = r ( t ) -Qr ( t ) S t into

Note r ( t ) ≥ 0 and ∥ AD -1 D 1 / 2 r ( t ) S t ∥ 1 = ∥ D 1 / 2 r ( t ) S t ∥ 1 . Hence, it leads to

At each local iterative t , by the active node condition 2 αϵ √ d u / (1 + α ) ≤ r ( t ) u , we have

Then the total run time of LOCGD presented in Algo. 4 is

Therefore, the total run time is at most T LOCGD := ∑ T -1 t =0 vol( S t ) ≤ 1+ α 2 αϵ . For estimating the bounds of T , by the Weierstrass product inequality [26] and Lemma B.5, we use the similar argument made in Lemma B.3 and continue to have

Note that each nonzero ˜ r ( T ) u has at least part of the magnitude from the push operation of an active node, say v u at time t ′ < T . This means each nonzero of D 1 / 2 r ( T ) satisfies

Hence, we have ∥ ˜ r ( T ) ∥ 1 ≥ 2 α (1 -α ) ϵ |I T | (1+ α ) 2 and T is further bounded as

The lower bound of 1 /ϵ , i.e., vol( S t ) /γ T ≤ 1 /ϵ , directly follows a similar strategy of previous proof by noticing that ∥ D 1 / 2 r ( t ) ∥ 1 is monotonically decreasing.

Remark B.6 . One may consider designing local methods based on Jacobi and Richardson's iterations. Indeed, these two methods have the same updates as standard GD. Recall the standard GD method to solve (4) is x ( t +1) = x ( t ) + r ( t ) , r ( t +1) = r ( t ) -Qr ( t ) . The Richardson's iteration is x ( t +1) = ( I -ω W ) x ( t ) + ω b , i.e., x ( t +1) = x ( t ) -ω ( Wx ( t ) -b ) . The optimal ω ∗ = 2 / ( λ min + λ max ) where λ min = 2 α/ (1 + α ) and λ max ≤ 2 / (1 + α ) . Hence one can choose ω = 1 ≤ ω ∗ [19]. It leads to x ( t +1) = x ( t ) + r ( t ) . One can get the same result for the Jacobi method.

C Local Chebyshev Method - LOCCH

C.1 Nonhomogeneous of Second-order Difference Equation

We begin by providing the solutions of the second-order nonhomogeneous equation as the following Lemma C.1 (Stevi´ c [47]) . The solution of the second-order nonhomogeneous difference equation

is characterized by the following two cases

̸

where ˆ λ 1 , ˆ λ 2 are two roots of λ 2 + pλ + q = 0 , and the summation follows convention ∑ 0 k =1 · = 0 . Based on the above lemma, we have the following corollary

Corollary C.2 (Second-order nonhomogeneous equation) . Given | a | ≤ 1 , the second-order nonhomogeneous equation has the following solution

Proof. The first and last two cases can be directly followed from Lemma C.1. Since ˆ λ 1 and ˆ λ 2 are the two complex roots of λ 2 -2 aλ + 1 = 0 , we write ˆ λ 1 = re iθ = r (cos θ + i sin θ ) and ˆ λ 2 = re -iθ = r (cos θ -i sin θ ) . It indicates

Since 1 > a 2 , ˆ λ 1 = a -i √ 1 -a 2 , and Re ( ˆ λ 1 ) 2 + Im ( ˆ λ 1 ) 2 = a 2 +(1 -a 2 ) = 1 , then r = 1 . Then θ = arccos( Re ( ˆ λ 1 )) = arccos( a ) , and sin( θ ) = Im ( ˆ λ 1 ) = √ 1 -a 2 . Finally, ˆ λ t 1 = e itθ = cos( tθ ) + i sin( tθ ) , and

Based on these, we have

Lemma C.3. For | λ i | ≤ 1 , i = 1 , 2 , . . . , n , equations y ( t +1) i -2 λ i y ( t ) i + y ( t -1) i = 0 have the following solutions

Furthermore, when y (1) i = λ i y (0) i for i = 1 , 2 , . . . , n , then solutions can be simplified as

Proof. The first part is a consequence of Corollary C.2 by letting f t = 0 . To see the second identity of this lemma, note that

Indeed, by expanding sin( θ ( t 1))

Hence, when y (1) = λ i y (0) , we have

Lemma C.4. Given t ≥ 1 , | λ i | ≤ 1 , the n second-order difference equations have the following solutions

Furthermore, with initial conditions y (1) = λ i y (0) , y ( t )

Proof. The first part is a consequence of Corollary C.2 by letting f t = h i,t . We prove the second part by considering three cases of λ i as

Case 1. When λ i = 1 , we have y ( t +1) i -2 y ( t ) i + y ( t -1) i = h i,t . For t ≥ 1 , the solution the above is

where the second equation is due to y (1) i = λ 1 y (0) i = y (0) i .

Case 2. When λ n = -1 ( G is a bipartite graph), we have y ( t +1) i +2 y ( t ) i + y ( t -1) i = h i,t . For t ≥ 1 , the solution is

Case 3. When | λ i | < 1 , and define θ i = arccos( λ i ) . We use a similar argument in Lemma C.3 and continue to have

C.2 Properties on Ratio of Chebyshev Polynomials

The next lemma presents the properties of Chebyshev polynomials.

Lemma C.5 (Chebyshev polynomial bound) . For t ≥ 1 , the Chebyshev polynomial of the first kind is defined recursively as

For t ≥ 1 , define δ t = T t -1 ( 1+ α 1 -α ) /T t ( 1+ α 1 -α ) , then

1. T t ( x = 1+ α 1 -α ) and δ t defines the following sequence

The closed-form δ 1: t can be upper bounded as

Note δ 1 = T 0 /T 1 = 1 /x = 1 -α 1+ α , the sequence { δ t } satisfies δ t < 1 , ∀ t ≥ 1 and

Denote δ j : t = ∏ t i = j δ i for t ≥ j ≥ 0 and set the default value δ j : j -1 = 1 for j ≥ 0 , then t k .

Proof. For the first item, let x = 1+ α 1 -α , use the Chebyshev equation, we have

For the second item, for all t ≥ 0 , if ξ = x + x -1 2 = 0 , it is well known that the T t can be rewritten as

̸

For our problem, recall we defined ˜ α = 1 - √ α 1+ √ α and using x = 1+ α 1 -α = 0 , one can verify that

and

For the third item, it is sufficient to show that δ t ≤ 1 x for all t ≥ 1 , for x = 1+ α 1 -α . This can be done recursively, since δ 1 = x and

For the last item, note when t ≥ 1 and k ≥ 1 , we have the following inequalities

where note ˜ α 2 +1 ˜ α 2 t +1 ∈ [1 , 2] for t ≥ k .

̸

C.3 Standard Chebyshev (CH) Method and Proof of Theorem C.6

This subsection introduces the standard Chebyshev algorithm. Our following theorem is to prove the runtime complexity of the Chebyshev polynomial iteration for solving Equ. (3).

Theorem C.6 (Standard CH) . For t ≥ 1 , consider the Chebyshev polynomials to solve Equ. (3) as where x (0) = 0 , x (1) = x (0) + r (0) and δ t +1 = ( 2 1+ α 1 -α -δ t ) -1 with δ 1 = 1 -α 1+ α . Assume ϵ < 1 /d s , then the residual has the following convergence bound

Let the estimate be ˆ π = D 1 / 2 x ( t ) , the the runtime of CH for reaching ∥ D -1 / 2 r ( t ) ∥ ∞ ≤ 2 αϵ 1+ α with ∥ D -1 (ˆ π -π ) ∥ ∞ ≤ ϵ guarantee is at most

Proof. Recall eigendecomposition of W = V Λ V ⊤ where V = [ v 1 , v 2 , . . . , v n ] and each v i is the eigenvector. For t ≥ 1 , the residual r ( t ) can be written as n second-order difference equations as

where each i -th element-wise equation of the above can be written as the following

Define V ⊤ r ( t ) = δ 1: t y ( t ) . Each component v ⊤ i r ( t ) is v ⊤ i r ( t ) = δ 1: t y ( t ) i where v ⊤ i r (0) := y (0) i by default. The above can be rewritten

̸

where ⇐⇒ follows from δ 1: t +1 = 0 . Note V ⊤ r (1) = 1 -α 1+ α V ⊤ V Λ V ⊤ r (0) = δ 1 Λ V ⊤ r (0) , we have

̸

where it follows from δ 1 = 0 . As y (1) = Λ y (0) , follow Equ. (24) of Lemma C.3, y ( t ) i has the solution where θ i = arccos( λ i ) . We can write down r ( t ) in terms of y ( t )

where Z t has two possible forms

Hence, ∥ Z t ∥ 2 ≤ 1 and ∥ Z t y (0) ∥ 2 ≤ ∥ y (0) ∥ 2 . We have

where the last inequality follows Lemma C.5 and note z (0) = V ⊤ r (0) . To meet the stop condition of { | r ( t ) u | < 2 αϵ √ d u / (1 + α ) , u ∈ V } = ∅ , it is sufficient to choose a minimal integer t such that

To see this, if the above inequality is satisfied, then note for any node u , we have

where all nodes are inactive. So, it gives t ln ( 1 - √ α 1+ √ α ) ≤ ln ϵ 2 by noticing ∥ b ∥ 2 = 2 α/ ((1 + α ) √ d s ) . It indicates t ≥ ⌈ ln 2 ϵ / ln ( 1+ √ α 1 - √ α ) ⌉ . As 1 ln(1+ x ) ≤ 1+ x x for x > 0 , it is sufficient to choose t

C.4 Residual Updates of LOCCH and Proof of Lemma 4.1

We propose the following local Chebyshev iteration procedure

Lemma C.7. For t ≥ 1 with initials x (0) = 0 and x (1) = x (0) + r (0) S 0 , the local Chebyshev iterative is the following

Our next Lemma is to expanding ( x ( t ) -x ( t -1) ) S t

Denote ∆ ( t ) = x ( t ) -x ( t -1) , then ∆ ( t ) = ∑ t -1 j =0 ( (1 + δ j : j +1 ) ∏ t -1 r = j +1 δ r : r +1 r ( j ) S j : t -1 ) , where δ 0:1 = 0 , S 0: t = S 0 ∩ S 1 ∩ · · · ∩ S t and δ j : j +1 = δ j δ j +1 . We have the following where S j,t ≜ S j : t -1 ∩ S t .

Proof. Recall we defined δ 0 = 0 so that δ 0:1 = 0 . We prove this lemma by induction.

For t = 1 , note δ 0:1 = 0 and the support of r (0) is S 0 = supp( r (0) ) , then

For t = 2 , note S 1:1 = S 1 and S 0:1 = S 0 ∩ S 1 by our notation, then

For t = 3 , one can build x (3) -x (2) based on x (2) -x (1) and recall S 2:2 = S 2 , S 1:2 = S 1 ∩ S 2

For t = 4 , we continue to have

By induction t ≥ 1 ,

where the convention notation ∑ 0 i =1 · = 0 and ∏ i j = i +1 · = 1 . To verify the inductive step, consider for t +1 , we have

To see the second equation, note

where recall we denote S j,t ≜ S j : t -1 ∩ S t .

Lemma C.8 (Local Chebyshev updates) . Given the updates of x ( t +1) as defined by LOCCH in (10) , we have the following local updates

Proof. We only need to show the second equation of (28). The residual of LOCCH updates is

where the last equality is due to (1 + δ t : t +1 ) Qr ( t ) = (1 + δ t : t +1 ) r ( t ) -2 δ t : t +1 Wr ( t ) by noticing (1 + δ t δ t +1 ) 1 -α 1+ α = 2 δ t +1 in Lemma C.5. Hence, we have the second equation. To see the noisy part, note by Lemma C.7

Lemma 4.1 (Residual updates of LOCCH) . Given t ≥ 1 , x (0) = 0 , x (1) = x 0 + r (0) S 0 . The residual r ( t ) of LOCCH defined in Equ. (28) satisfies where

Proof. We first decompose the residual equation in (28) as

Define V ⊤ r ( t ) = δ 1: t y ( t ) and V ⊤ r (0) = δ 1:0 y (0) = y (0) by default. Then we have

Note V ⊤ r (1) = δ 1 y (1) = V ⊤ δ 1 Wr (0) = δ 1 Λ y (0) , which indicates y (1) = Λ y (0) . Then, by the Lemma C.4, each y ( t ) i has the solution

t

-

k

1

=1

j

k

=0



r

k

=

j

+1

(

-

1

1 +

α

α

)

V ⊤ r ( t ) = δ 1: t Z t V ⊤ r (0) ∑ ∑  ∏ Note 1 + δ j : j +1 = 2 δ j +1 1+ α 1 -α , j ≥ 1 , then (1 + δ j : j +1 ) ∏ k r = j +1 ( δ r : r +1 ) = 2 1+ α 1 -α δ j +1: k δ j +1: k +1 . Then, we have where the last term can be expanded as

.

.

.

+

δ

k

+2:

t

H

k,t

(1 +

δ

j

:

j

+1

)

(

δ

r

:

r

+1

)

I

-

Λ

V

⊤

r

(

j

S

)

j,k



.

Here, we denote δ k +1: k = 1 . The final iterative update is

C.5 Convergence of LOCCH and Proof of Theorem 4.2

Corollary C.9. Let β k be lower bound of residual reduction satisfies ∥ u k,t ∥ 2 ≤ β k ∥ r ( k ) ∥ 2 , then the upper bound of ∥ r ( t ) ∥ 2 can be characterized as

where y 0 = y 1 = ∥ r 0 ∥ 2 .

Proof. Since ∥ u k,t ∥ 2 ≤ β k ∥ r ( k ) ∥ 2 , the final iterative updates (29) can be bounded as

where t = 0 , 1 , . . . , T . These T +1 (including a trivial one where ∥ r (0) ∥ 2 ≤ ∥ r (0) ∥ 2 ) inequalities shown in Equ. (31) form a system of linear inequality matrix as

where ( Z L ) tk = 2 δ k +1: t ( t -k ) β k for t = 2 , 3 , . . . , T and k = 1 , 2 , . . . , t -1 . Assume that N ∈ R T × T is a strictly lower triangular matrix, then we know the established formula ( I + N ) -1 = I + ∑ T -1 k =1 ( -1) k N k . Hence, we have the following

Given that ( I -Z L ) -1 ≥ 0 , then we obtain an upper-bound of

It leads to the following new second-order difference equation

where the initial value of z 0 = ∥ r (0) ∥ 2 . Following the argument in Theorem 1 of Golub & Overton [18], we construct a second-order homogeneous equation for z t as

where Equ. (32) is obtained by the difference between the second equation and the third equation. Similarly,

where Equ. (33) is obtained by the difference of the first two. Hence, Equ. (32) -(33) gives us where two initials are z 0 = r (0) 0 and z 1 = δ 1 (1 + β 0 ) r (0) 2 . Let z t = δ 1: t ˆ z t , then

where two initials are ˆ z 0 = z 0 = ∥ r (0) ∥ 0 and ˆ z 1 = (1 + β 0 ) ∥ r (0) ∥ 2 . We finish the proof by setting ∏ t -1 j =0 (1 + β j ) y t = ˆ z t .

Remark C.10 . Key points of the above proof strategy largely follow from Golub & Overton [18]. However, different from the original technique, we generalize the strategy to a parameterized version.

Lemma C.11. Given β j ≥ 0 , the following second-order difference equation has the following solution

where y t +1 -2 y t + y t -1 (1+ β t -1 )(1+ β t ) = 0 with y 0 = x 0 and y 1 = x 1 / (1 + β 0 ) .

Proof. Assume x t = ( -1 2 ) t ∏ t -1 j =0 ( -2(1 + β j )) y t . Then, following the equation, we have

̸

Since β j ≥ 0 , the term ∏ t j =0 ( -2(1 + β j )) = 0 , we divide it on both sides to have

Hence, it is simplified into y t +1 -2 y t + y t -1 (1+ β t -1 )(1+ β t ) = 0 . To make a simplification on x t , we prove the lemma.

Theorem 4.2 (Runtime bound of LOCCH) . Given the configuration θ = ( α, ϵ, s, G ) with α ∈ (0 , 1) and ϵ ≤ 1 /d s and let r ( T ) and x ( T ) be returned by LOCCH defined in (10) for solving Equ. (3) . For t ≥ 1 , the residual magnitude ∥ r ( t ) ∥ 2 has the following convergence bound

where y t is a sequence of positive numbers solving y t +1 -2 y t + y t -1 / ((1 + β t -1 )(1 + β t )) = 0 with y 0 = y 1 = ∥ r (0) ∥ 2 . Suppose the geometric mean β t ≜ ( ∏ t -1 j =0 (1 + β j )) 1 /t of β t be such that β t = 1 + c √ α 1 - √ α where c ∈ [0 , 2) . There exists a real implementation of (9) such that the runtime T LOCCH is bounded by

Proof. The convergence bound of r ( t ) directly follows from Corollary C.9. Since we assume that there exists c ∈ [0 , 2) such that ∏ t -1 j =0 (1 + β j ) ≤ ( 1 + c √ α 1 - √ α ) t . Then multiplying both sides by ˜ α t , we have

Then we have

Since β t = ( ∏ t -1 j =0 (1 + β j ) ) 1 /t , and by using 1+ x x ≥ 1 ln(1+ x ) and letting x = 1+ √ α ( 1 - √ α ) β t -1 > 0 , then t can be lower bounded further by

Since we assumed β t = (1 + c √ α 1 - √ α ) , which means 1 ≤ ¯ β t = (1 + c √ α 1 - √ α ) , so β t ∈ [ 1 , 1+ √ α 1 - √ α ] . Then, we find such an upper bound of t so that LOCCH converges.

C.6 Implementation of LOCCH

We present the implementation of LOCCH as follows: Recall the sequence δ t +1 = ( 2 1+ α 1 -α -δ t ) -1 , t = 1 , 2 , . . . with δ 1 = 1 -α 1+ α . Denote ˜ x ( t ) ≜ x ( t ) -x ( t -1) , ∆ ( t ) := (1 + δ t : t +1 ) r ( t ) + δ t : t +1 ˜ x ( t ) , we have

When t = 0 , we have x (0) = 0 , r (0) = b , ˜ x (0) = 0 , ∆ (0) = r (0) .

When t = 1 , we have x (1) = r (0) S 0 , r (1) = 1 -α 1+ α W ∆ (0) S 0 , ˜ x (1) = r (0) S 0 , ∆ (1) = (1 + δ 1:2 ) r (1) + δ 1:2 ˜ x (1) .

When t ≥ 1 , we can recursively calculate the following vectors

Therefore, at per-iteration, we only need to save sub-vectors ∆ S t and ∆ S t -1 and update x locally.

D Local Heavy-Ball Method - LOCHB

D.1 Standard HB and Proof Theorem D.2

Lemma D.1 (The standard HB updates) . The updates x ( t ) and r ( t ) of the HB method for solving Equ. (4) can be written as

The residual updates can be rewritten as a second-order homogeneous equation

where y ( t ) is such that r ( t ) = ˜ α t V y ( t ) , t ≥ 0 with y (0) = V ⊤ r (0) = V ⊤ b .

Proof. We follow the standard Polyak's heavy-ball method [40] as

where ∇ f ( x ( t ) ) = Qx ( t ) -b and η α = 4 / ( √ 2 / (1 + α )+ √ 2 α/ (1 + α )) 2 = 2(1+ α ) / (1+ √ α ) 2 = 1 + ˜ α 2 and η β = ( √ 2 / (1 + α ) -√ 2 α/ (1 + α )) 2 / ( √ 2 / (1 + α ) + √ 2 α/ (1 + α )) 2 = ˜ α 2 . Hence, it leads to the following updates

Inserting

then x ( t +1) = 2˜ α Wx ( t ) -˜ α 2 x ( t -1) +(1 + ˜ α 2 ) b and since

So

Using r ( t ) = ˜ α t V y ( t ) , t ≥ 0

As W = V Λ V ⊤ and V ⊤ = V -1 is orthogonal matrix, we continue to have

Theorem D.2 (Convergence analysis of Heavy-Ball (HB)) . To solve the minimization problem in Equ. (4) , we propose the following standard HB updates as

where the initial condition is x (0) = 0 , r (0) = b , x (1) = x (0) + Γ r (0) , r (1) = b -Qx (1) . Then there exists a constant τ such that the total iteration complexity to reach the stop condition { u : | r u | ≤ ϵd u , u ∈ V} = ∅ is where C t = 1 if Γ = Q -1 ( I -˜ α W ) (ideal case); C t = max { 1+˜ α -1 √ 1 -λ 2 2 , 1 + (1 + ˜ α -1 ) t } if Γ = 0 (1 ˜ α )(1+ α ) is not bi-partite graph).

(practical case); and C t = 2 √ 1 -λ 2 2 if Γ = -2 I ( G

Proof. Recall W = V Λ V ⊤ and then V ⊤ r ( t +1) = 2˜ α Λ V ⊤ r ( t ) -˜ α 2 V ⊤ r ( t -1) . By Lemma D.1, we have

where we obtained n second-order difference equations

Follow the Lemma C.3, Equ. (23) has the solution

where in the case of | λ i | < 1 . We consider the three cases of Γ

Ideal Case: We can eliminate t in (34), when y (1) = Λ y (0) , we get y (1) i = λ i y (0) i , and then y ( t ) i can be simplified into

In this case, Γ needs to be Γ = Q -1 ( I -˜ α W ) . Therefore, we have

Practical Case: Just letting x (1) = x (0) = 0 , we have ˜ α y (1) = y (0) , then

Non-bipartite graph Case : When the graph is non-bipartite, we can eliminate t , as the following: We choose Γ = τ I , we have

We have the following relations

To make t disappear when λ i = 1 , we need y (0) i = y (1) i , or

In this case, we have

To make sure the algorithm stops when the stop condition is met, it is enough for

This means C t ∥ r (0) ∥ 2 ˜ α t ≤ ϵ , which leads to the following

D.2 Residual Updates of LOCHB and Proof of Theorem D.6

Lemma D.4. Let the local heavy-ball method be defined as

where x (0) = 0 , x (1) = Γ r (0) and Γ = diag (Γ 1 , Γ 2 , . . . , Γ n ) is initial step size matrix. We have the following expanding sequence

Furthermore, we have the following sequence

where we denote S i,t ≜ S i : t -1 ∩ S t .

Proof. We assume all nonzeros in b are active nodes at time t = 0 and t = 1 , i.e., S 0 = r (0) = supp( b ) . The local updates can be expressed as

For t ≥ 1 , we can expand x ( t +1) -x ( t ) as the following

Note S 0 = supp( r (0) ) , then r (0) S 1: t = r (0) S 0: t , we continue to have

The rest follows readily.

Lemma D.5 (The nonhomogeneous difference equation) . Given y (1) = Λ y (0) , equations

have the following solutions

where

Proof. This directly follows from Lemma C.4.

Theorem D.6 (Representation of r ( t ) for LOCHB) . Given t ≥ 1 , x (0) = 0 and x (1) = Γ r (0) S 0 . The residual of r ( t ) of LOCHB satisfies

So we can write the updates of LOCHB as

Write V ⊤ r ( t ) = ˜ α t y ( t ) , and note

Hence,

Then, from Lemma D.5

so expanding the error term

which when simplified, gives the relation proposed.

D.3 Convergence of LOCHB of Proof of Theorem D.8

Then the upper bound of ∥ r ( t ) ∥ 2 can be characterized as where y t +1 -2 y t + y t -1 / ((1 + β t -1 )(1 + β t )) = 0 where y 0 = y 1 = ∥ r (0) ∥ 2 .

Proof. Since ∥ u k,t ∥ 2 ≤ β k ∥ r ( k ) ∥ 2 , then given the final iterative updates (29)

and since we can bound

where t = 0 , 1 , . . . , T . The rest just follows a similar strategy shown in Corollary C.9. We have the following inequalities

The above analysis finally leads to τ t +1 -2(1 + β t )˜ ατ t + ˜ α 2 τ t -1 = 0 . So for

τ t +1 -2˜ ατ t + ˜ α τ t -1 = 2˜ αβ t τ t we have

Following the same strategy in Corollary C.9, we obtain the upper bound.

Theorem D.8 (Convergence of LOCHB) . Let the geometric mean of β t be β t ≜ ∏ t -1 j =0 (1 + β j ) 1 /t . Then the upper bound of ∥ r ( t ) ∥ 2 can be characterized as where y t +1 -2 y t + y t -1 / ((1 + β t -1 )(1 + β t )) = 0 where y 0 = y 1 = ∥ r (0) ∥ 2 . Assume that there exists a constant c ∈ [0 , 2) such that β t ≤ 1 + c √ α 1 - √ α . Then the total number of iterations can be bounded as

Then the total runtime T is bounded by where ˜ O hides ln( y t /ϵ ) .

Proof. The first part follows from Corollary of D.7. The rest follows the same strategy of LOCCH as in Theorem 4.2. (See also Theorem 4.2.)

D.4 Implementation of LOCHB

We present the implementation of LOCHB as follows: Recall the updates of LOCHB is x ( t +1) = x ( t ) +(1 + ˜ α 2 ) r ( t ) S t + ˜ α 2 ( x ( t ) -x ( t -1) ) S t , r ( t +1) = 2˜ α Wr ( t ) -˜ α 2 r ( t -1) . The corresponding local updates are

where if we choose x (0) = x (1) = ˜ x (1) = 0 , r (0) = r (1) = b and ∆ (0) = 0 .

E Instances of Sparse Linear Systems

E.1 Table of Popular Graph-induced Linear Systems

This section presents most commonly used graph-induced linear system as the following

where Q is the generalized version of the perturbed normalized graph Laplacian matrix with perturbation parameter σ > 0 , and b is a sparse vector. A typical example of Q = I -1 -α 1+ α D -1 / 2 AD -1 / 2 with Λ = I and b = 2 α D -1 / 2 e s / (1 + α )

The detailed parameters are:

1. W = ˜ D -1 / 2 ˜ A ˜ D -1 / 2 and ˜ A = I + A is the adjacency matrix defined on G ( V , E ) by adding self-loops for all nodes, and ˜ D = I + D is defined as the augmented degree matrix by adding self-loops. α ∈ (0 , 1) and usually α < 0 . 5 , The I -(1 -α ) W is the perturbed augmented normalized Laplacian with perturbed parameter α .

3. x = α ( I -(1 -α ) AD -1 ) -1 e s This is the standard Personalized PageRank vectors widely used for graph embeddings and graph neural network designing [7]. It is also used for decoupling for large-scale GNNs [56].

2. Q = D -1 / 2 ( D -1 -α 2 ( D + A ) ) D -1 / 2 = α I + 1 -α 2 L > 0 and Q = 1 -α 1+ α D -1 / 2 AD -1 / 2 . This is known as the lazy random-walk version of PPR vectors.

4. Graph kernel computation for online learning. Each computed vector serves as semi-supervised learning feature vectors [24] or as online node labeling learning vectors [41, 57]. Note the target linear system when σ = 1

Hence, we have Λ = λ n D -1 + I n , σ = 1 , b = 2 λ D -1 / 2 e s .

F Experimental Details and Missing Results

F.1 Datasets and Preprocessing

Following Leskovec et al. [34], we treat all 17 graphs as undirected with unit weights. We remove selfloops and keep the largest connected component when the graph is disconnected. After preprocessing, the graphs range from 169 , 343 nodes in ogbn-proteins to 111 , 059 , 433 in ogbn-papers100M, as presented in Table 3.

F.2 Problems Settings and Baseline Methods

For solving Equation (3), we randomly select 50 source nodes s from each graph. The damping factor is fixed at 0 . 1 , i.e., α = 0 . 1 for all experiments, and it varies within the range { 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 15 , 0 . 2 , 0 . 25 , 0 . 3 } for others. The ϵ is chosen from the range [ 2 α/ ((1 + α ) d s ) , 10 -4 /n ] .

For solving the local clustering problem, we follow the greedy strategy from Andersen et al. [2], where a local cluster is identified by examining the top magnitudes in PPR vectors. Specifically, we denote the boundary of S as ∂ ( S ) = { ( u, v ) ∈ E : u ∈ S , v / ∈ S} . The conductance of S is defined as

0

1

2

×

10

8

-

5

.

0

-

2

.

5

0

.

0

ln

‖

ˆ

x

-

x

∗

‖

1

as-skitter

0

2

×

10

8

-

2

0

cit-patent

0

1

×

10

7

-

4

-

2

0

com-dblp

0

1

2

×

10

10

-

2

-

1

com-friendster

0

2

4

×

10

8

-

2

-

1

com-lj

0

5

×

10

8

-

1

.

0

-

0

.

5

com-orkut

0

2

4

×

10

7

-

3

-

2

-

1

ln

‖

ˆ

x

-

x

∗

‖

1

com-youtube

0

1

×

10

10

-

5

0

ogb-mag240m

0

1

2

×

10

8

-

1

.

5

-

1

.

0

-

0

.

5

ogbl-ppa

0

1

2

×

10

7

-

4

-

2

0

ogbn-arxiv

0

1

2

×

10

8

-

1

.

5

-

1

.

0

-

0

.

5

ogbn-mag

0

1

×

10

10

-

2

-

1

ogbn-papers100M

0

5

# operations

10

8

-

3

-

2

-

1

ln

‖

ˆ

x

-

x

∗

‖

1

ogbn-products

0

1

# operations

10

8

-

0

.

4

-

0

.

2

ogbn-proteins

0

.

0

2

.

5

5

.

0

# operations

10

8

-

4

-

2

0

soc-lj1

0

1

2

# operations

10

8

-

1

.

5

-

1

.

0

-

0

.

5

soc-pokec

0

1

# operations

10

9

-

1

.

5

-

1

.

0

-

0

.

5

wiki-en21

0

2

# operations

10

7

-

2

-

1

wiki-talk

×

×

×

×

×

×

0

25

50

75

Speedup

as-skitter

/

LOCGD

SOR

/

LOCSOR

HB

/

LOCHB

CH

/

LOCCH

0

50

100

cit-patent

0

50

100

com-dblp

0

50

com-lj

0

100

200

com-orkut

0

20

40

60

Speedup

com-youtube

0

50

100

150

ogbl-ppa

0

20

40

60

ogbn-arxiv

0

25

50

75

ogbn-mag

0

100

200

300

ogbn-products

10

-

8

10

-

5

10

-

2

/epsilon1

0

50

100

Speedup

ogbn-proteins

10

-

8

10

-

5

10

-

2

/epsilon1

0

50

soc-lj1

10

-

8

10

-

5

10

-

2

/epsilon1

0

50

100

150

soc-pokec

10

-

8

10

-

5

10

-

2

/epsilon1

0

50

100

150

wiki-en21

10

-

8

10

-

5

10

-

2

/epsilon1

0

5

10

wiki-talk

The goal of local clustering is to obtain PPR vectors using these local methods and then apply clustering algorithms to find clusters with low conductance. For the sorting process, given the approximate PPR vector ˜ π , we sort D -1 / 2 ˜ π in decreasing order of magnitudes. Let the ordered nodes be v 1 , v 2 , . . . , v t ; the local clustering algorithm iteratively checks the conductance reduction by v 1 , v 2 , · · · , v k where k = 1 , 2 , . . . , t , and after completing all checks, it returns a subset v 1 , v 2 , . . . , v k ′ that has the minimal conductance among all examined subsets.

Parameter settings of baselines. For the local ISTA method [13], the precision parameter is set to ˆ ϵ = 0 . 5 for all experiments. According to the algorithm's description of ISTA, the corresponding ρ value is given by ϵ/ (1 + ˆ ϵ ) . For LOCSOR, the parameter ω is calculated as 2(1 + α ) / (1 + √ α ) 2 . For the local FISTA, as demonstrated in [22], we adopt the same settings as for ISTA and follow its implementation guidelines. We also include preliminary results on ASPR [37]. The algorithm incorporates a parameter, ˆ ϵ , to control the number of iterations in the nested Accelerated Projected Gradient Descent (APGD). We adjust ˆ ϵ from low precision, ˆ ϵ = 0 . 1 /n , to high precision, ˆ ϵ = 10 -4 /n , to ensure the identification of a good approximation.

For our experiment, we used a server powered by an Intel(R) Xeon(R) Gold 5218R CPU, which features 40 cores (80 threads). The system is equipped with 256 GB of RAM.

F.3 Full results of Fig. 15 4 5 6

In all 15 graphs, we set α = 0 . 1 and ϵ = 0 . 1 /n . For each of the testing graphs, we randomly select 50 nodes and run LOCSOR and CGM.

0

2

×

10

6

-

4

-

2

ln

‖

x

(

T

)

-

x

‖

1

as-skitter (

/epsilon1

=1.12e-09)

LocSOR

LocHB

LocCH

0

2

4

×

10

7

-

4

-

2

cit-patent (

/epsilon1

=1.12e-09)

0

2

4

×

10

6

-

4

-

2

com-dblp (

/epsilon1

=1.12e-09)

0

.

0

0

.

5

1

.

0

×

10

8

-

4

-

2

com-lj (

/epsilon1

=1.12e-09)

0

1

2

×

10

8

-

4

-

3

com-orkut (

/epsilon1

=1.12e-09)

0

2

×

10

7

-

4

-

2

ln

‖

x

(

T

)

-

x

‖

1

com-youtube (

/epsilon1

=1.12e-09)

0

1

2

×

10

7

-

4

-

3

ogbl-ppa (

/epsilon1

=1.12e-09)

0

.

0

2

.

5

5

.

0

×

10

6

-

4

-

2

ogbn-arxiv (

/epsilon1

=1.12e-09)

0

.

0

0

.

5

1

.

0

×

10

8

-

4

-

3

ogbn-mag (

/epsilon1

=1.12e-09)

0

2

×

10

7

-

4

-

2

ogbn-products (

/epsilon1

=1.12e-09)

0

5

Operations

10

6

-

4

.

5

-

4

.

0

-

3

.

5

ln

‖

x

(

T

)

-

x

‖

1

ogbn-proteins (

/epsilon1

=1.12e-09)

0

.

0

0

.

5

1

.

0

Operations

10

8

-

4

-

2

soc-lj1 (

/epsilon1

=1.12e-09)

0

.

0

0

.

5

1

.

0

Operations

10

8

-

4

-

3

-

2

soc-pokec (

/epsilon1

=1.12e-09)

0

1

2

Operations

10

8

-

4

-

3

wiki-en21 (

/epsilon1

=1.12e-09)

0

2

4

Operations

10

7

-

6

-

4

-

2

wiki-talk (

/epsilon1

=1.12e-09)

×

×

×

×

×

Fig. 8 presents all speedup tests on 15 datasets. It is evident that these standard linear solvers can be localized effectively.

0

1

2

3

×

10

8

-

6

-

4

-

2

ln

‖

x

(

T

)

-

x

‖

1

as-skitter (

/epsilon1

=1.58e-09)

LocSOR

LocHB

LocCH

ISTA

FISTA

CGM

APPR

0

.

0

2

.

5

5

.

0

7

.

5

×

10

8

-

5

-

4

-

3

-

2

cit-patent (

/epsilon1

=7.13e-10)

0

2

4

×

10

7

-

4

-

2

com-dblp (

/epsilon1

=8.46e-09)

0

.

0

0

.

5

1

.

0

×

10

9

-

5

-

4

-

3

-

2

com-lj (

/epsilon1

=6.71e-10)

0

1

2

×

10

9

-

5

-

4

-

3

-

2

com-orkut (

/epsilon1

=8.73e-10)

0

1

2

×

10

8

-

5

-

4

-

3

-

2

-

1

ln

‖

x

(

T

)

-

x

‖

1

com-youtube (

/epsilon1

=2.36e-09)

0

2

×

10

8

-

4

.

5

-

4

.

0

-

3

.

5

-

3

.

0

-

2

.

5

ogbl-ppa (

/epsilon1

=4.66e-09)

0

2

4

6

×

10

7

-

5

-

4

-

3

-

2

ogbn-arxiv (

/epsilon1

=1.58e-08)

0

.

0

0

.

5

1

.

0

×

10

9

-

5

-

4

-

3

-

2

ogbn-mag (

/epsilon1

=1.38e-09)

0

.

0

0

.

5

1

.

0

×

10

9

-

6

-

5

-

4

-

3

-

2

ogbn-products (

/epsilon1

=1.12e-09)

0

1

2

3

Operations

10

8

-

5

.

0

-

4

.

5

-

4

.

0

-

3

.

5

ln

‖

x

(

T

)

-

x

‖

1

ogbn-proteins (

/epsilon1

=2.02e-08)

0

.

0

0

.

5

1

.

0

1

.

5

Operations

10

9

-

5

-

4

-

3

-

2

-

1

soc-lj1 (

/epsilon1

=5.54e-10)

0

.

0

0

.

5

1

.

0

Operations

10

9

-

5

-

4

-

3

-

2

soc-pokec (

/epsilon1

=1.64e-09)

0

2

4

Operations

10

9

-

4

.

5

-

4

.

0

-

3

.

5

-

3

.

0

-

2

.

5

wiki-en21 (

/epsilon1

=4.32e-10)

0

2

4

Operations

10

8

-

6

-

4

-

2

wiki-talk (

/epsilon1

=1.12e-09)

×

×

×

×

×

Fig. 9 presents the missing results on the estimation error reduction for 15 datasets. Compared with the global solver CGM, all local methods show significant speedup in the early stages. To compare our three local solvers, we zoom in on our results and present them in Fig. 10. Empirically, LOCSOR is the fastest algorithm when the parameter ω is chosen optimally.

0

1

2

×

10

7

-

3

.

5

-

3

.

0

-

2

.

5

-

2

.

0

ln

‖

x

(

T

)

-

x

‖

1

(Node 0,

/epsilon1

= 1

/n

)

APPR

LocHB

LocCH

LocSOR

0

2

×

10

7

-

3

.

00

-

2

.

75

-

2

.

50

-

2

.

25

(Node 1,

/epsilon1

= 1

/n

)

0

2

×

10

7

-

3

.

0

-

2

.

8

-

2

.

6

-

2

.

4

(Node 2,

/epsilon1

= 1

/n

)

0

2

×

10

7

-

3

.

00

-

2

.

75

-

2

.

50

-

2

.

25

(Node 3,

/epsilon1

= 1

/n

)

0

2

×

10

7

-

3

.

0

-

2

.

5

-

2

.

0

(Node 4,

/epsilon1

= 1

/n

)

0

2

Operations

10

9

-

4

-

3

-

2

ln

‖

x

(

T

)

-

x

‖

1

(Node 0,

/epsilon1

= 1

/m

)

0

1

2

Operations

10

9

-

3

.

5

-

3

.

0

-

2

.

5

(Node 1,

/epsilon1

= 1

/m

)

0

1

2

Operations

10

9

-

3

.

5

-

3

.

0

-

2

.

5

(Node 2,

/epsilon1

= 1

/m

)

0

2

Operations

10

9

-

3

.

5

-

3

.

0

-

2

.

5

(Node 3,

/epsilon1

= 1

/m

)

0

2

Operations

10

9

-

4

.

0

-

3

.

5

-

3

.

0

-

2

.

5

-

2

.

0

(Node 4,

/epsilon1

= 1

/m

)

×

×

×

×

×

0

2

4

×

10

7

-

3

-

2

ln

‖

x

(

T

)

-

x

‖

1

(Node 0,

/epsilon1

= 1

/n

)

APPR

LocHB

LocCH

LocSOR

0

5

×

10

7

-

3

.

0

-

2

.

5

(Node 1,

/epsilon1

= 1

/n

)

0

5

×

10

7

-

3

.

0

-

2

.

8

-

2

.

6

-

2

.

4

(Node 2,

/epsilon1

= 1

/n

)

0

5

×

10

7

-

3

.

2

-

3

.

0

-

2

.

8

-

2

.

6

-

2

.

4

(Node 3,

/epsilon1

= 1

/n

)

0

.

0

2

.

5

5

.

0

×

10

7

-

3

.

5

-

3

.

0

-

2

.

5

-

2

.

0

-

1

.

5

(Node 4,

/epsilon1

= 1

/n

)

0

.

0

0

.

5

1

.

0

Operations

10

9

-

4

-

3

-

2

ln

‖

x

(

T

)

-

x

‖

1

(Node 0,

/epsilon1

= 1

/m

)

0

1

2

Operations

10

9

-

4

.

0

-

3

.

5

-

3

.

0

-

2

.

5

-

2

.

0

(Node 1,

/epsilon1

= 1

/m

)

0

2

Operations

10

9

-

3

.

5

-

3

.

0

-

2

.

5

(Node 2,

/epsilon1

= 1

/m

)

0

2

Operations

10

9

-

4

.

0

-

3

.

5

-

3

.

0

-

2

.

5

(Node 3,

/epsilon1

= 1

/m

)

0

.

0

0

.

5

1

.

0

Operations

10

9

-

4

-

3

-

2

(Node 4,

/epsilon1

= 1

/m

)

×

×

×

×

×

0

1

2

3

×

10

7

-

2

.

5

-

2

.

4

-

2

.

3

-

2

.

2

-

2

.

1

ln

‖

x

(

T

)

-

x

∗

‖

1

ogbl-ppa

(

α

= 0

.

005)

0

2

4

6

8

×

10

6

-

3

.

5

-

3

.

0

-

2

.

5

-

2

.

0

ogbl-ppa

(

α

= 0

.

1)

0

.

0

0

.

5

1

.

0

1

.

5

×

10

6

-

5

-

4

-

3

-

2

ogbl-ppa

(

α

= 0

.

25)

LocGD

LocSOR

LocHB

LocCH

0

1

2

Operations

10

7

-

2

.

8

-

2

.

6

-

2

.

4

-

2

.

2

ln

‖

x

(

T

)

-

x

∗

‖

1

ogbn-products

(

α

= 0

.

005)

0

.

0

0

.

5

1

.

0

Operations

10

7

-

5

-

4

-

3

ogbn-products

(

α

= 0

.

1)

0

2

4

Operations

10

6

-

6

-

5

-

4

-

3

ogbn-products

(

α

= 0

.

25)

×

×

×

0

500

1000

-

3

-

2

-

1

0

Conductance

as-skitter

APPR

LocCH

LocHB

LocSOR

ISTA

FISTA

0

500

1000

1500

-

1

.

0

-

0

.

5

0

.

0

cit-patent

0

100

200

300

-

2

-

1

0

com-dblp

0

50

100

-

3

-

2

-

1

0

com-lj

0

2000

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

com-orkut

0

50

-

2

-

1

0

Conductance

com-youtube

0

250

500

750

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

ogbl-ppa

0

50

100

150

-

2

-

1

0

ogbn-arxiv

0

2000

4000

-

1

.

0

-

0

.

5

0

.

0

ogbn-mag

0

20

40

60

-

2

-

1

0

ogbn-products

0

500

1000

1500

Support Size

-

1

.

0

-

0

.

5

0

.

0

Conductance

ogbn-proteins

0

50

100

Support Size

-

4

-

2

0

soc-lj1

0

200

400

Support Size

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

soc-pokec

0

2500

5000

7500

Support Size

-

1

.

0

-

0

.

5

0

.

0

wiki-talk

0

1000

2000

3000

Support Size

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

wiki-en21

F.4 Results on local clustering

G Related work

Many graph applications [2, 7, 29, 15, 27, 36, 30, 44, 25, 45, 51, 48] only require solving Equ. (1) approximately. The reasons could be either the most energies of π are among a small set of nodes forming small subgraphs, or one wants to study large graphs by checking them locally. Given a graph G with n nodes and m edges, there are two main types of iterative solvers for Equ. (1) as follows:

Standard iterative methods. Methods for solving linear systems have been well-established over the past decades (see textbooks of Saad [43], Golub & Van Loan [19], Young [55]). The fastest linear solver for solving the symmetrized version of Equ. (1) is the Conjugate Gradient Method (CGM) with runtime

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

Number of operations needed

×

10

10

-

3

.

0

-

2

.

5

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

ln

‖

ˆ

x

-

x

∗

‖

1

LocCH

CGM

complexity ˜ O ( m/ √ α ) where m is the number of edges in the graph. It costs Θ( m ) to access the entire graph at each iteration; hence, it is much slower than local solvers, as demonstrated in Fig. 15. The symmetric diagonally dominant (SDD) solvers advance CGM further to have complexity ˜ O ( m log c n log(1 /ϵ )) [31, 46]. Anikin et al. [4] considered the PageRank problem and proposed an algorithm with runtime depending on O ( n ) . This paper focuses on local algorithms where the goal is to avoid the dominant factor m or n by avoiding the full Qx operation.

Local algorithms. Local solvers, in contrast to standard counterparts, leverage the fact that the energy of π lives in a small portion of the graph and hence do not require O ( m ) or O ( n ) per iteration. They are advantageous for huge-scale graphs demonstrated in Fig. 15. Andersen et al. [2] used APPR to obtain an approximate of π for local clustering. Quite similar algorithms were developed in Berkhin [6] ( bookmark-coloring algorithm) and Kloster & Gleich [28] (Gauss-Southwell procedure).

Under the same stopping condition as APPR, Fountoulakis et al. [13] demonstrated that APPR is equivalent to coordinate descent via variational characterization, with a runtime of ˜ O (1 / ( αϵ )) using ISTA where the monotonicity and conservation properties remain. Hence, it is nature to ask whether ˜ O (1 / ( √ αϵ )) could be achieved by FISTA [5] in Fountoulakis & Yang [12]. However, the difficulty is that FISTA violates the monotonicity property where the volume accessed of per-iteration cannot be bounded properly. To overcome this, Martínez-Rubio et al. [37] proposed a nested accelerated projected gradient descent (APGD) and gradually expanding solutions so that the monotonicity property still holds. However, nested APGD requires solving subproblems accurately, which in practice may be cumbersome if the precision requirement of the inner problem is too stringent. All current local methods rely on some monotonicity property of variables to guarantee locality, which does not exist in most accelerated frameworks; thus, developing an accelerated method that is guaranteed to preserve intermediate variable sparsity remains challenging.

It is worth mentioning that local methods are also closely related to sublinear time and local computational algorithms [42, 1]. From the optimization perspective, the equivalence between Gauss-Seidel and coordinate descent has been considered [49, 35, 39, 50] but does not focus on local analysis.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the main claims made by the paper, including its key contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The accelerated algorithms, such as LOCCH and LOCHB, may exhibit instability when α is small. This limitation arises due to the inherent constraints of global methods.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are clearly stated.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided our code for the review process and will make it publicly available upon publication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All datasets used in this study are publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https:// nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All parameter settings of our methods and baselines are included.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For most of our results, we report the standard error over 50 random sampling nodes. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: For our experiment, we used a server powered by an Intel(R) Xeon(R) Gold 5218R CPU, which features 40 cores (80 threads). The system is equipped with 256 GB of RAM.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics. The authors have thoroughly reviewed and adhered to the guidelines, ensuring that all aspects of their work align with ethical standards.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The advancements in accelerated algorithms like LOCCH and LOCHB can significantly enhance computational efficiency in various applications, contributing to faster and more effective solutions in fields such as data analysis, machine learning, and optimization.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: None.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: None.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: None.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human subjects involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the study of local algorithms for graph clustering which is an important problem in the field of graph data analysis. In particular this paper is considers the task of computing Personalized Page Rank (PPR) vectors for a given graph. In this problem the algorithm is given a graph in the form of its adjacency and degree matrices, the goal is to approximate the Personalized Page Rank vector for a given starting vertex and dampening factor $\\alpha$ up to precision $\\epsilon$ without accessing the entire graph. The classical algorithm of Andersen, Chung and Lang runs in time $O(1/\\alpha \\epsilon)$, which independent of the graph size. The central question posed by subsequent works is whether the dependence on $\\alpha$ can be improved to $1/\\sqrt{\\alpha}$. The main contribution of the paper is to propose a new algorithmic framework based on the locally evolving set process. Under this framework they are able to implement existing algorithms such as Andersen et al.'s APPR algorithm as well as localized implementation of standard gradient descent. They are also able to develop localized versions of chebyshev and heavy ball methods that do achieve the $1/\\sqrt{\\alpha}$ dependence for some fixed constant value of $\\epsilon$. Finally they show that on several large scale graphs, their new localized chebyshev and heavy ball methods do outperform APPR and related methods empirically.\n\nSTRENGTHS:\nThe main strengths of this paper are to develop a new algorithmic framework that can not only encompass existing algorithms but lead to the development of better ones that overcome previously known limitations for designing local graph clustering algorithms. They also back their theoretical analysis with the practical implementation of their method which is also shown to be superior to previous algorithms.\n\nWEAKNESSES:\nOne weakness is that the paper is only able to obtain a quadratic improvement in the dependence on the parameter $\\alpha$, obtained by the local implementation of the Chebyshev and Heavy-ball method, only for a value of $\\epsilon$ and not for all.\n\nQUESTIONS:\nOne question is that what is the core reason for not obtaining convergence result for accelerated methods for all $\\epsilon>0$.\n\nLIMITATIONS:\nAuthors have addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

comments: SUMMARY:
This paper uses the evolving set procedure to give a local PageRank algorithm whose dependence on \alpha (the reset probability) is \sqrt{\alpha}.

It proposes accelerated local iterative methods with coefficients given by Chebyshev iteration. The convergence of this algorithm in both graph theoretic and general sparse linear systems settings are analyzed in detail. Discussions of the relations between this method and other local iterative algorithms are also given in detail.

The method was implemented and tested on a range of graphs, mostly coming from social networks. This includes two large ones with edges in the billions. On moderate ranges of \alpha (reset probability), the experiments show significant speedups (factor of about 3) and convergences (factor of 10) on most graphs.

STRENGTHS:
Local algorithms are widely used in graph analytics. The question studied is natural, and has been proposed before.

The method is theoretically well-founded, and has significant technical depth.

The experiments are thorough and well documents, and clearly demonstrate the advantages of this method in multiple parameter regimes.

WEAKNESSES:
The gains only kick in at a relatively large number of steps: it's not clear to me that these are the parameter regimes in which local algorithms actually get used.

Ideally for the empirical works I'd also like to see comparisons of downstream tasks and effects on overall accuracies (e.g. F-1 score), but the paper itself has already covered a lot of ground.

QUESTIONS:
Is the dependence on \epsilon optimal? Aka. have methods with \sqrt{\eps} (or even \log(1 / \eps)) dependences been ruled out?

LIMITATIONS:
yes, limitations have been addressed, and are entirely theoretical w.r.t. some graph parameter regimes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-05

comments: SUMMARY:
This paper considers the approximate personalized page rank. Classical results for this problem have a runtime that is linear in $1/\alpha\epsilon$ where $\alpha$ is the damping factor and $\epsilon$ is the error parameter. The authors show that APPR is simply a local variant of Gauss-Seidel Successive Overrelaxation. Using this connection, the authors derive new run time bounds for APPR and also propose a new algorithm based on Gradient Descent. The execution time for both these are, in the worst-case, identical to the previous bounds. However, they are more sensitive to the state of execution of the algorithms (depend on the active nodes) and seem to mirror the actual performance of these algorithms. Also, under certain assumptions, they improve the worst-case execution time.

STRENGTHS:
The paper addresses an important problem, provides deeper insights into an existing algorithm, provides a new algorithm and also reanalyzes the algorithm in a more fine-grained way. All of this is done via connection to GSSOR which seems to be new.

I find the result quite interesting. However, I am not very familiar with recent work on personalized page rank.  For this reason, I recommend accepting but with a low confidence.

WEAKNESSES:
NA

QUESTIONS:
Can you explain how weak/strong are the assumptions that you make to achieve the improvement for the local Chebyshev method? I couldn't quite gauge the usefulness of this result.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13