Title: Efficient Learning of Linear Graph Neural Networks via Node Subsampling
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph neural networks

Random sampling

Regression
Abstract: Graph Neural Networks (GNNs) are a powerful class of machine learning models with applications in recommender systems, drug discovery, social network analysis, and computer vision. One challenge with their implementation is that GNNs often take large-scale graphs as inputs, which imposes significant computational/storage costs in the training and testing phases. In particular, the message passing operations of a GNN require multiplication of the graph adjacency matrix $A \in \mathbb{R}^{n \times n}$ and the data matrix $X \in \mathbb{R}^{n \times d}$, and the $O(n^2 d)$ time complexity can be prohibitive for large $n$. Thus, a natural question is whether it is possible to perform the GNN operations in (quasi-)linear time by avoiding the full computation of $A X$. To study this question, we consider the setting of a regression task on a two-layer Linear Graph Convolutional Network (GCN). We develop an efficient training algorithm based on (1) performing node subsampling, (2) estimating the leverage scores of $A X$ based on the subsampled graph, and (3) performing leverage score sampling on $A X$. We show that our proposed scheme learns the regression model observing only $O(nd\epsilon^{-2}\log n)$ entries of $A$ in time $O(nd^2 \epsilon^{-2}\log n)$, with the guarantee that the learned weights deviate by at most $\epsilon$ under the $\ell_2$ norm from the model learned using the entire adjacency matrix $A$. We present empirical results for regression problems on real-world graphs and show that our algorithm significantly outperforms other baseline sampling strategies that exploit the same number of observations.

Efficient Learning of Linear Graph Neural Networks via Node Subsampling

Abstract

Graph Neural Networks (GNNs) are a powerful class of machine learning models with applications in recommender systems, drug discovery, social network analysis, and computer vision. One challenge with their implementation is that GNNs often take large-scale graphs as inputs, which imposes significant computational/storage costs in the training and testing phases. In particular, the message passing operations of a GNN require multiplication of the graph adjacency matrix A ∈ R n × n and the data matrix X ∈ R n × d , and the O ( n 2 d ) time complexity can be prohibitive for large n . Thus, a natural question is whether it is possible to perform the GNN operations in (quasi-)linear time by avoiding the full computation of AX . To study this question, we consider the setting of a regression task on a two-layer Linear Graph Convolutional Network (GCN). We develop an efficient training algorithm based on (1) performing node subsampling, (2) estimating the leverage scores of AX based on the subsampled graph, and (3) performing leverage score sampling on AX . We show that our proposed scheme learns the regression model observing only O ( ndε -2 log n ) entries of A in time O ( nd 2 ε -2 log n ) , with the guarantee that the learned weights deviate by at most ε under the ℓ 2 norm from the model learned using the entire adjacency matrix A . We present empirical results for regression problems on real-world graphs and show that our algorithm significantly outperforms other baseline sampling strategies that exploit the same number of observations.

1 Introduction

Graph Neural Networks (GNNs) have gained popularity as a powerful machine learning method for graph-structured data. By learning rich representations of graph data, GNNs can solve a variety of prediction tasks on graphs [1-7]. GNNs have delivered impressive results across many different areas, including social network analysis [8], bioinformatics [9-11], and recommendation systems [12].

Given their remarkable performance, being able to train GNNs efficiently is an important task. However, training GNNs can be quite challenging in the context of large-scale graphs, which impose significant computational costs. In particular, the message passing scheme in GNNs requires, for each node, summing up the feature vectors of all neighboring nodes into one feature vector. For a multi-layer Graph Convolutional Network (GCN) [3], the layer-wise propagation rule is

where H ( ℓ ) denotes the feature representation at the ℓ th layer, with H (0) = X (i.e., the data matrix consisting of nodes' features of an input graph G ); W ( ℓ ) denotes the weight matrix; σ ( · ) is a non-linear activation function like the ReLU; and A denotes the adjacency matrix of G .

If the graph has n nodes and the feature dimension is d , computing the matrix multiplication AX requires O ( n 2 d ) time and can be prohibitive in big data settings. A natural question is whether it is possible to train a GCN to avoid the quadratic complexity scaling with n .

Another motivation for avoiding the full computation of AX is applications where the adjacency matrix A is not fully known a priori, and it must be learned via node/edge queries. For example, in large social networks (the current number of social media users is over 4 . 89 billion [13]), one may need to access the adjacency matrix A by querying the adjacency list of specific nodes. As another example, certain biological networks must be learned through the physical probing of pairwise interactions, which may make obtaining the entire adjacency matrix A prohibitively expensive. For example, the mapping of the neural network of living organisms (such as the connectome of C. elegans [14]) requires physical probing of the connectivity between neurons.

A natural approach to avoid the quadratic complexity of computing AX is via the subsampling of the graph G [15, 6, 16, 17]. If the resulting subgraph is sparse enough, AX can be computed very efficiently. But how many entries of A need to be observed in order to guarantee that the message passing step AX can be computed accurately enough in the context of GNNs? What kinds of graph subsampling strategies are amenable to theoretical performance guarantees on the training of GNNs?

We consider the setting of a regression task to be learned via a GCN. Let A ∈ R n × n be the weighted adjacency matrix of a graph G = ( V, A ) with | V | = n . Each node v i ∈ V has an associated feature vector of dimension d ( < n ) and a label, denoted by ( x i , y i ) ∈ R d × R , which can also be represented as a n × d data matrix X and label vector y . Training the GCN corresponds to minimizing the loss on the training data { ( x i , y i ) } n i =1 , where W denotes the GCN network weights, and f GCN ( W,A,X ) denotes a feed-forward computation of a GCN's output. As a first step to studying whether (2) can be solved accurately on a sparse subgraph of G , we focus on a simple linear GCN, where there is no non-linearity σ ( · ) . Specifically, the feed-forward output is given by f linear GCN ( w , A, X ) := AX w , where we use w ∈ R d (instead of W ∈ R 1 × d ) to indicate that the learnable parameters are in the form of a vector. Hence, our goal is to solve

Note that one can view this optimization problem as a graph-weighted linear regression problem .

The setting of linear regression provides natural suggestions for a subsampling strategy. It is known that leverage score sampling allows one to solve the linear regression problem with (1 + ε ) accuracy using only O ( dε -2 log n ) subsampled rows of the data matrix X and in time O ( nd 2 + d 3 ) 1 [18-22]. Nevertheless, to apply this strategy to (3), one would need to first compute AX , requiring O ( n 2 d ) time, so that the row leverage scores of AX can then be computed. This motivates us to propose a two-step approach that (1) performs node subsampling to obtain an approximate observation of AX and estimate the leverage scores of the rows of the augmented matrix [ AX | -y ] ∈ R n × ( d +1) and (2) performs leverage score sampling on the rows of [ AX | -y ] using the estimated scores.

This two-step approach is illustrated in Figure 1. In order to obtain an estimator of AX , our sampling scheme in the first step builds O (log n ) rank1 matrices by sampling the same amount of columns of A and the corresponding rows of X . The key idea behind this approach is taking a random subset of nodes and propagating their feature vectors to their neighbors. We provide a spectral approximation guarantee for the resulting estimate of AX and an approximation guarantee for the leverage scores of the augmented matrix [ ̂ AX | -y ] computed on this estimate. In the second step, we adopt the standard leverage score sampling for sampling rows of A , using the leverage score estimates of [ AX | -y ] obtained from the first step. With O ( dε -2 log n ) sampled rows of ˜ A and ˜ y , the training algorithm then computes ˜ AX and uses it to solve the regression problem. We show that our proposed scheme learns the regression model with learned weights deviated by at most ε from those with full information, by observing only O ( ndε -2 log n ) entries of A , and in time O ( nd 2 ε -2 log n ) .

On real-world benchmark datasets, we demonstrate the performance improvements of the two proposed schemes over other baseline sampling schemes via numerical simulations.

<latexi sh

1_b

64="Z

5

wo

Q30

2Ey

9WF+fK

Vq

>A

DGX

c

J

8

d

m

P

zgT

N

R

j

I

u

CL

n

p

O

r

k

v

/

Y

7

U

M

B

S

H

<latexi sh

1_b

64="D0

Nf

TI

S

5c

z

/3

YPKCnQ

>A

B8H

V

E

Ur

q

9L

G

k

wm7

gov

2+

j

d

J

X

F

p

W

u

Z

M

O

y

R

<latexi sh

1_b

64="95UQky/LG

Of

VI7

X

jNz

Aqg

>

DE3

c

R

J

p

Pr

M

K

n

BC

u20

W

8md

+

Z

FY

v

w

S

o

H

T

<latexi sh

1_b

64="B79p2vF

H

A

D8

uLO

Y0X

C

>

c

V

M

WG

k

T

Sj

Uo

q

gy

QJR

I

Zw

E

+

P

m

3

zn

/d

5

N

f

r

K

<latexi sh

1_b

64="

MXuEvqOzCy8m2

cn

R37VS

N

>A

D

L

w

F

W

r9

J

G

U

oZ

kP

K

I

p

B

T

j

H

d

/0

5

+

f

g

Y

Q

<latexi sh

1_b

64="OQy

X/nqAB

57M

LkSd

E

Ug

>

DH

c

VJ

9

wF

W8r

j

KC

Y

3

GI

N

0

T

z

f

R

+

v

Z

m

oP

2

u

p

<latexi sh

1_b

64="PUO

3

z

kYXrLmW

j

nq

u

o

>A

DE

c

V

w

FN

9R

JG

g

K

/p

S5

CHIMv

Q

f

7

d2

B

+

y

Z

8

T

0

<latexi sh

1_b

64="v

2A5qw

r

PgfNpU

>

C

c

VL

Q

GM7

0

u

T

y

J

O

F

RWn

8

Z

KE

oI

j

B

/

H

m

+

Y

S

9

<latexi sh

1_b

64="Eq

H/8

DQB

vTY

9w g5WV

fm

>A

K3

c

M

F

X

jk

R

G

p

y

0

u

Z

o

r

L

z

I

+

P

Sn

O7

N

2

C

U

<latexi sh

1_b

64="

f

X

Gjz8

N

9q

IU

SoM

T

>A

D

3

c

VLd

QwF

7

v

+

Ey

C

/

k

u

WZ2

pO

J0

R

K

P

r

mH

n

5

g

B

Y

<latexi sh

1_b

64="Y+vAQS8u

R

HqPVwkC

J

E

>

D

X

c

Ld

F

Nf rp

WB

T

y

oU

Z

2

n

3

/

O

09

j

K

I

g

G

m

7

<latexi sh

1_b

64="c nV

DKJRMY

w

Nv kQHfq

3

>A

X

L

F

WG

2y

Um

j

E

Z

+

O

o

I9

T

z

d

C

7B

/

g

p

5

0

8

P

r

S

u

<latexi sh

1_b

64="

MXuEvqOzCy8m2

cn

R37VS

N

>A

D

L

w

F

W

r9

J

G

U

oZ

kP

K

I

p

B

T

j

H

d

/0

5

+

f

g

Y

Q

<latexi sh

1_b

64="RQ

r

zW

0S

D

c

>A

CX

VL

w

F

9

JGm

3Y

g

K

/p ko

B

O

j

v2

7

f

+

n

y

P

u

8

T

M

U

E

H

<latexi sh

1_b

64="wyRA

EYJO+

ok

Mmgz

Q/L

K8

>

DGX

c

V

FNW

r9

Z

Iu

U

Sp

C

T

7

P

2

f

3

d

v

q

H

5

j

0

n

B

<latexi sh

1_b

64="fd pu/YGV

AXI

r

CR

FP

8

>

D

3

c

J

9

w

W

j

5

S

H

o

0

mE

Q

U7B

+

k

2

y

K

L

z

N

Z

TM

O

q

g

v

n

linear GNN

architecture

˜

A

̂

AX

=

̂

A

Step 2: Use leverage score estimate for training

̂

y

×

X

×

˜

A

sample                         rows

via leverage score sampling

sample                                      's

p

j

A

:

j

×

X

j

:

w.p.

O

(

/epsilon1

-

2

log

n

)

perform a regression task

O

(

d/epsilon1

-

2

log

n

)

̂

X

y

/lscript

i

([

̂

AX

| -

y

])

Notation: Throughout the paper, we use A = ( a ij ) ∈ R n × n ≥ 0 to denote the graph adjacency matrix, where each entry value is nonnegative and bounded (i.e., 0 ≤ a ij ≤ M for some constant M > 0 ). We write A i : and A : j to indicate the i th row and j th column of the matrix A respectively. In addition, X ∈ R n × d denotes the data (design) matrix, where the i th row in X , x i , corresponds to the node vector at the i th node v i in the graph G . We also assume that the absolute value of each entry value of X is bounded by M (i.e., | X ij | ≤ M ). We denote vectors by a lowercase bold letter (e.g., x ∈ R n ), and by default, all the vectors will be in column format, hence X = ( x ⊤ 1 ; . . . ; x ⊤ n ) ⊤ , where ( · ) ⊤ denotes the transpose. We indicate the i th entry of x with either [ x ] i or x i , and we use [ x ] S to denote the concatenation of x i s for all i ∈ S . We also let [ n ] := { 1 , 2 , . . . , n } . We denote by ⟨ x , y ⟩ := ∑ i x i y i ∈ R the inner product between x and y . Unless otherwise mentioned, we use ∥ v ∥ for the ℓ 2 -norm for vector v . We let ∥ A ∥ := max ∥ v ∥ =1 ∥ A v ∥ denote the operator norm of a matrix, and ∥ A ∥ F := √ ∑ i,j a 2 ij denote the Frobenius norm. We denote the standard basis vector by e i . We write Bern( p ) for a Bernoulli distribution with parameter p . Lastly, we denote by nnz ( A ) the number of non-zero entries in matrix A .

Related Work: We provide a detailed discussion of related works in Appendix A.

2 Motivation: Leverage Score Sampling for Linear Regression

First, we briefly describe key results on leverage score sampling that will be relevant to our discussion. We refer to [23] for a detailed discussion on the subject, and we provide key proofs in Appendix D. Definition 1. The leverage score of the i th row of X is ℓ i ( X ) := x ⊤ i ( X ⊤ X ) † x i , where ( · ) † denotes the Moore-Penrose pseudoinverse.

Intuitively, a row's leverage score measures how important it is in composing the row space of X . If a row has a component orthogonal to all the other rows, its leverage score is 1 . Removing it would decrease the rank of X , completely changing its row space. The leverage scores are the diagonal entries of the projection matrix X ( X ⊤ X ) -1 X ⊤ , which can be used to show that ∑ n i =1 ℓ i ( X ) = rank( X ) . The following alternative characterizations of ℓ i ( X ) will be useful.

Proposition 2. The leverage score can be alternatively computed as ℓ i ( X ) = max v ( x ⊤ i v ) 2 ∥ X v ∥ 2 2 .

Proposition 1. Let X = U Σ V ⊤ be the singular value decomposition (SVD) of X , where U ∈ R n × d , Σ ∈ R d × d and V ∈ R d × d . Then ℓ i ( X ) = ∥ U i : ∥ 2 2 , where U i : is the i th row of U .

Through the use of leverage scores, it is possible to approximately solve a linear regression task using a subset of the data points [18-22]. In particular, leverage score sampling allows the spectral approximation of a matrix.

Definition 2. A matrix ̂ X ∈ R n × d is an ε -spectral approximation of X ∈ R n × d if, for all v ∈ R d , (1 -ε ) · ∥ X v ∥ 2 ≤ ∥ ̂ X v ∥ 2 ≤ (1 + ε ) · ∥ X v ∥ 2 . (4) Suppose we are given a data matrix X = ( x ⊤ 1 ; . . . ; x ⊤ n ) ⊤ and a vector u = ( u 1 , . . . , u n ) of leverage score overestimates of X (i.e., ℓ i ( X ) ≤ u i , ∀ i ∈ [ n ] ). We create a matrix ˜ X by including the i th row

of X in ˜ X with probability p i = min[1 , cu i ε -2 log n ] for some constant c , scaled by 1 / √ p i . The following is a slight modification of Lemma 4 in [21] (which we prove in Appendix D):

Lemma 1. With probability at least 1 -n -Ω(1) , for any v , we have

Notice that the expected number of rows in ˜ X is c ∥ u ∥ 1 ε -2 log n . Hence, as long as ∥ u ∥ 1 = O ( d ) , we only need to sample O ( dε -2 log n ) rows of X to obtain a good spectral approximation.

Now suppose we want to solve the OLS problem min w 1 2 ∥ y -X w ∥ 2 2 where y ∈ R n is the response vector corresponding to X . The optimal solution is given by w ∗ = ( X ⊤ X ) -1 X ⊤ y . Exactly computing w ∗ requires O ( nd 2 + d 3 ) time.

An alternative approach using leverage score sampling would be as follows. First, we note that the OLS objective can be rewritten using ∥ y -X w ∥ = ∥ [ X | -y ] · [ w 1 ] ∥ . If we now have leverage score overestimates u 1 , . . . , u n for the augmented matrix [ X | -y ] ∈ R n × ( d +1) , we can construct the subsampled matrix [ X S | -y S ] ∈ R | S |× ( d +1) (with rows rescaled by 1 / √ p i ). Then, by Lemma 1, a solution to would satisfy ∥ y S -X S ˜ w ∥ 2 ≤ ∥ y S -X S w ∗ ∥ 2 ≤ (1 + ε ) ∥ y -X w ∗ ∥ 2 .

The exact leverage scores of X can be computed in time O ( nd 2 ) (see Algorithm 3 in Appendix B) and, since | S | = O ( dε -2 log n ) , the reduced linear system (5) can be solved in time O ( d 3 ε -2 log n ) . The overall complexity for solving the OLS problem in this way is given by O ( nd 2 + d 3 ε -2 log n ) . Notice that there is no real acceleration obtained by this procedure. But also notice that here we are using the exact leverage scores for sampling while, from Lemma 1, we know that this approach works as long as we have good leverage score overestimates.

3 Efficient Training of a Linear GCN via Leverage Scores

In this section, we introduce an efficient training algorithm that approximately solves the graphweighted regression task (3) by observing only O ( ndε -2 log n ) entries of A , and in time O ( nd 2 ε -2 log n ) . The necessity for introducing the efficient training algorithm comes from the O ( n 2 d ) time complexity of computing AX via matrix multiplication.

Motivated by the discussion in Section 2 for standard linear regression, a natural approach is to attempt to compute leverage score overestimates for AX efficiently without the full computation of AX . Our proposed algorithm uses the following three steps to approximately solve the graph-weighted linear regression problem.

ESTIMATELEVERAGESCORES ( A,X ) (See Algorithm 1 and Algorithm 2)

LEVERAGESCORESAMPLING ( A, { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } ) (See Algorithm 4 in Appendix B)

REGRESSIONSOLVER ( ˜ AX, ˜ y ) (See Algorithm 5 in Appendix B)

In ESTIMATELEVERAGESCORES ( A,X ) , a node subsampling technique is used to build an estimate ̂ AX of AX with an approximation guarantee. The estimate ̂ AX can then be used to produce provably good overestimates of the leverage scores of the augmented matrix [ AX | -y ] . Next, LEVERAGESCORESAMPLING ( A, { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } ) uses these overestimates to produce a matrix ˜ A and scaled labels ˜ y consisting of a reduced number of rows of A and y respectively. Using Lemma 1, we then show that ˜ AX provides a spectral approximation for AX , which allows approximately solving the regression problem (3) using REGRESSIONSOLVER ( ˜ AX, ˜ y ) . We note that none of the algorithmic procedures requires fully computing AX and the end-to-end algorithm has an O ( n log n ) time dependence on n .

Algorithm 1 ESTIMATELEVERAGESCORES ( A,X ) via Uniform Sampling

Input: Adjacency matrix A ∈ R n × n , data matrix X ∈ R n × d , budget B , threshold ε > 0 Output: Leverage score overestimates for AX

3: ̂ AX ← ∑ n j =1 ̂ A : j X j : ▷ O ( Bd ) 4: S ← LEVERAGESCORE ([ ̂ AX | -y ]) ▷ See Alg. 3 in App. B; ˜ O ( nnz ([ ̂ AX | -y ]) + d 3 ) 5: { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } ← (1+ ε ) 2 (1 -ε ) 2 · S

1: Draw I j ∼ Bern( p ) , ∀ j ∈ [ n ] independently, where p = min [ B n , 1 ] 2: ̂ A : j X j : ← I j p A : j X j : , ∀ j ∈ [ n ]

6: return ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ]

3.1 First Stage: Uniform Sampling of A : j X j :

In order to efficiently estimate the leverage scores of AX , a key observation we make is that AX can be decomposed as a sum of n rank1 matrices as AX = ∑ n j =1 A : j X j : , where A : j and X j : denote j th column of the adjacency matrix A and j th row of the data matrix X . Notice that one can view A : j X j : as the effect of node j 's feature vector propagated to other nodes after the one-step message passing operation. Hence, a natural approximation strategy is to take a subset of these rank1 matrices. First, we consider a uniform sampling strategy that samples node indices j uniformly at random and computes A : j X j : . See Algorithm 1 for the detailed procedure.

Given a budget B (the total number of nodes that should be observed), the algorithm draws sampling indicator variables I j ∼ Bern( p ) independently with p = min[ B/n, 1] . For j = 1 , . . . , n , we build

and then set ̂ AX := ∑ j ̂ A : j X j : . Notice that AX is an unbiased estimate of AX , because for all i, j ,

In addition, one can readily see that B rank1 matrices are sampled in expectation, and, with high probability, the actual number of sampled rank1 matrices is less than (1 + α ) B for α > 0 .

Next, we present a spectral approximation guarantee for ̂ AX . In order to establish that result, we make the following mild assumption on the density of the input graph. Assumption 1 . We assume that the input graph G satisfies

Strictly speaking, (7) is not just an assumption about the density of G , as it takes into consideration the interaction between A and X . Nevertheless, this assumption would be expected to hold in cases where the graph G is dense. In particular, since ∥ AX ∥ ≥ d -1 / 2 ∥ AX ∥ F , Assumption 1 holds if ∥ AX ∥ F ≥ n 3 / 2 d . Assuming that A has δn 2 non-zero entries and that entries of X are drawn i.i.d. from some distribution with positive second moment β 2 , independently of A , we have ∥ AX ∥ F ≈ √ δn 2 dβ 2 = βδ 1 / 2 n 3 / 2 d 1 / 2 . Treating d as a constant and rescaling X by a constant, we have that the assumption holds. More generally, if d is a constant and

∣ ∣ { ( i, j ) ∈ [ n ] × [ d ] : e ⊤ i AX e j ≥ c 1 n }∣ ∣ ≥ c 2 n for constants c 1 and c 2 , then the assumption holds (after a constant rescaling of the data matrix). The following stronger assumption will be needed to establish guarantees for the leverage score estimates: Assumption 2 . For all ( i, j ) ∈ [ n ] × [ d ] , e ⊤ i AX e j ≥ c 1 n for some constant c 1 .

Notice that if the graph G is instead sparse, the problem we focus on due to the computation of AX is less severe since one can efficiently compute AX in time that depends on nnz ( A ) .

The proof of the following theorem is in Appendix E.

Theorem 1. Under Assumption 1, if we run Algorithm 1 with budget B = Cε -2 log n for large enough C > 0 , the estimate AX satisfies with probability at least 1 -n -Ω(1) .

Given the spectral approximation guarantee in Theorem 1, we expect the leverage scores computed on the augmented matrix [ ̂ AX | -y ] to be close to the leverage scores of [ AX | -y ] . Since we want an overestimate of the leverage scores, we use the the multiplicative constant ( 1+ ε 1 -ε ) 2 to obtain

The following lemma, whose proof is in Appendix F, asserts that ̂ ℓ i ([ AX | -y ]) is indeed an overestimate of ℓ i ([ AX | -y ]) .

Lemma 2. Under Assumption 2, Algorithm 1 with budget B = Cε -2 log n for large enough C > 0 outputs leverage score overestimates satisfying

with probability at least 1 -n -Ω(1) .

3.2 Second Stage: Leverage Score Sampling and Approximate Regression

The second step of the proposed algorithm exploits the leverage score estimates ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] to perform the standard leverage score sampling. With Lemma 1, one can readily obtain a (1 + ε ) -approximation guarantee for the graph-weighted linear regression task.

Theorem 2. Under Assumption 2, suppose we run Algorithm 1 with budget B = Cε -2 log n for large enough C > 0 , to obtain { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } . Subsequently, if we run LEVERAGESCORESAMPLING ( A, { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } ) , we obtain ˜ A ∈ R O ( dε -2 log n ) × n and ˜ y ∈ R O ( dε -2 log n ) such that REGRESSIONSOLVER ( ˜ AX, ˜ y ) provides ˜ w satisfying

Theorem 2 captures our main contribution. By only observing O ( d log n ) rows of A , we can obtain a reduced adjacency matrix ˜ A (and corresponding label vector ˜ y . Running REGRESSIONSOLVER ( ˜ AX, ˜ y ) then yields a solution ˜ w with an approximation guarantee. We refer to detailed descriptions of LEVERAGESCORESAMPLING ( A, { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } ) and REGRESSIONSOLVER ( ˜ AX, ˜ y ) in Appendix B.

Sample complexity and run-time: We note that the sample complexity is O ( ndε -2 log n ) . First, ESTIMATELEVERAGESCORES ( A,X ) exploits O ( ε -2 log n ) columns of A and the same number of rows of X , thus accessing O ( nε -2 log n ) entries of A and O ( dε -2 log n ) of X . In addition, LEVERAGESCORESAMPLING ( A, { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } ) exploits O ( dε -2 log n ) rows and thus uses O ( ndε -2 log n ) entries. Hence, the total number of entry observations is O ( ndε -2 log n ) .

For the time complexity, the run time of ESTIMATELEVERAGESCORES ( A,X ) is O ( ndε -2 log n ) ; that of LEVERAGESCORESAMPLING ( A, { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } ) is O ( nd 2 ε -2 log n ) . Finally, we note that the run time of REGRESSIONSOLVER ( ˜ AX, ˜ y ) is O ( nd 2 + d 3 ) 2 . Hence, the total run time is O ( nd 2 ε -2 log n ) given the setting that we consider has d < n .

Extension to the more-than-two-layer case: Note that without non-linearities, considering a multi-layer network is equivalent to performing additional matrix multiplication by A to the output of the previous layer. Since our proposed scheme yields a spectral approximation guarantee for matrix multiplication, an analysis similar to that of Algorithm 1 should be possible for this extension to obtain a (1 + ε ) L -approximation guarantee, where L denotes the number of layers. Since the approximation guarantee comes with high probability, a union bound could be used to obtain a high probability guarantee for the multi-layer case (since the order or error for each approximate matrix multiplication is the same).

4 Variance Minimization via Data-dependent Random Sampling

As stated in the previous section, Algorithm 1 produces an unbiased estimate of AX . Yet the uniform sampling strategy used in Algorithm 1 does not take the variance of the estimator into consideration. This motivates us to develop an estimator of AX that further reduces the estimator variance while maintaining unbiasedness. Due to the unbiasedness of ̂ AX , the total variance of the estimator coincides with the mean square error (MSE): Var( AX ) = E ∥ AX -AX ∥ 2 F = MSE ( AX ) .

̂ ̂ ̂ In order to reduce the estimator variance Var( ̂ AX ) , we consider a data-dependent sampling probability p j that depends on the j th column of A and the j th row of X . As we view A : j X j : as capturing the impact of node j 's features after a one-step message passing operation, one way to set the sampling probability for node j is to take the norm of A : j X j : into account. Specifically, we set

Algorithm 2 describes the detailed procedure, where the only distinction relative to Algorithm 1 is the choice of p j . We claim that this sampling probability choice minimizes the total variance (as well as the MSE) of the estimator. First notice that the total variance can be expressed as where ( i ) follows from the fact that

and ( ii ) follows from using the fact that ∑ j p j = 1 ; and ( iii ) follows from Cauchy-Schwarz inequality. Notice that, in order for Cauchy-Schwarz to hold with equality, we must have p k proportional to ∥ A : k ∥∥ X k : ∥ . This implies that (11) is an optimal choice of the sampling probabilities.

As it turns out, the same theoretical guarantees obtained for Algorithm 1 hold in this case. See Appendix G for the detailed proof. While the theoretical performance guarantee is the same, in settings where the budget B is very small, the data-dependent sampling probabilities p j lead to a better estimate AX , which improves the overall performance of the algorithm.

̂ One important observation is that it is not obvious how to use the data-dependent sampling probabilities p j while avoiding the an O ( n 2 ) computation complexity. In particular, a naive computation of ∥ A j : ∥ , j ∈ [ n ] would require observing all n 2 entries of A . However, standard sampling techniques could be used in order to also estimate ∥ A j : ∥ , j ∈ [ n ] fairly accurate, and use those estimates to compute the sampling probabilities p j , j ∈ [ n ] .

Input: Adjacency matrix A ∈ R n × n , data matrix X ∈ R n × d , budget B , threshold ε > 0

3: ̂ AX ← ∑ n j =1 ̂ A : j X j :

5 Empirical Results

We support our theoretical findings via numerical experiments 3 to validate the performance of Algorithm 1 and Algorithm 2 in a non-asymptotic setting.

Dataset and evaluation methods: We consider benchmark datasets from the Open Graph Benchmark (OGB) [29], Stanford Network Analysis Project (SNAP) [30], and House dataset [31]. We defer detailed explanations of the datasets to Appendix H. For evaluation methods, we compute the mean squared error (MSE), wall-clock run-time, and peak memory usage of our two proposed schemes and five baselines: (1) REGRESSIONSOLVER ( AX, y ) with fully known A and X ; (2) REGRESSIONSOLVER ( ˜ AX, y ) with parts of rows of A , say ˜ A , obtained from sampling rows of A uniformly at random; (3) REGRESSIONSOLVER ( ˜ AX, y ) with the exact leverage score sampling for sampling rows of A to obtain partial ˜ A ; (4) REGRESSIONSOLVER ( ˜ AX, ˜ y ) with Algorithm 1; (5) REGRESSIONSOLVER ( ˜ AX, ˜ y ) with Algorithm 2; (6) REGRESSIONSOLVER ( ˜ AX, ˜ y ) with a sampling scheme inspired by GraphSage [6] that selects the neighborhood of feature aggregation for each node; and (7) REGRESSIONSOLVER ( ˜ AX, ˜ y ) with a sampling scheme motivated by GraphSaint [32] that selects a subgraph of the original graph. We note that (6) and (7) are not the tools themselves [6, 32], but graph subsampling strategies inspired by these tools.

Results and analysis:

Mean squared error: Figure 2 plots the mean squared error with respect to the budget allocation. First, we observe that Algorithm 1 and Algorithm 2 have a similar performance to the one with exact leverage scores sampling, and the performance of the two proposed schemes tends to be similar to that of REGRESSIONSOLVER with full ( AX, y ) , as the budget increases. Particularly seen from Figure 2a, the error reduction on Algorithm 2 from Algorithm 1 is 84% when the budget is at 3%. Even for the sparse datasets, as seen from Figure 2b and Figure 2c, the corresponding improvements of 86% and 67% exist when the budget is at 3% and 7% respectively. We also observe that the error decrease for Algorithm 2 over Algorithm 1; for instance Figure 2b demonstrates the existence of 91% improvement at 5% observation budget. On the other hand, we observe that other subsampling approaches-including those employed in uniform sampling, GraphSage, and GraphSaint-do not perform the graph-weighted regression task as well, especially in the low-budget regime.

Extension to nonlinear GCN: While our theoretical results are for linear GCNs, we also assess the efficacy of our proposed algorithm on nonlinear GCNs. Using the ReLU activation function and one hidden layer, Figure 2d plots the mean squared error as a function of the observation budget, shown as a percentage of the number of observed nodes in the graph. Interestingly, we observe that running the non-linear GCN with uniformly sampled ( AX, y ) fails to perform the graph-weighted regression task, especially in the low-budget regime. On the other hand, we observe that running the non-linear GCN with our proposed sampling technique has a similar performance to the one with exact leverage scores sampling, and the performance of the two proposed schemes tends to be similar to that of Regression with exact ( AX, y ) , as the budget increases. Particularly, the error reduction on Algorithm 1 from Regression with uniformly sampled ( AX, y ) is 60% when the budget is at 5%.

(a) Linear GCN on ogbl-ddi dataset [29] and generated according to the Cauchy distribution.

(c) Linear GCN on house dataset [33].

X (b) Linear GCN on ego-Facebook dataset [30] and X generated by the Cauchy distribution.

(d) Non-linear GCN on house dataset [33].

Wall-clock time: In Table 1, we provide a run-time comparison for the end-to-end training process for a regression task. We compare the wall-clock time of performing a regression task with full AX computation and that with Algorithm 1 that uses partial observations of A and X . The results show that our proposed scheme requires orders of magnitude less wall-clock time for large-scale graphs than the regression with exact AX computation. In particular, for the ogbn-arxiv dataset [29], our algorithm runs about 40x faster than the regression with the exact computation of AX .

Peak memory usage: We also demonstrate the efficacy on the memory usage of our proposed algorithms. See Table 3 in Appendix H for details.

6 Concluding Remarks

Motivated by the prohibitive computational/storage costs of running GNNs with large-scale graphs, we considered the problem of subsampling nodes' information for performing a regression task.

Our main contribution lies in providing an efficient sampling algorithm for learning linear GNNs with a large number of nodes with a theoretical guarantee. Specifically, for a two-layer linear GNN with adjacency matrix A ∈ R n × n and data matrix X ∈ R n × d , we show that it is possible to learn the model accurately while observing only O ( nd log n ) entries of A . This in turn yields a run-time of O ( nd 2 log n ) , avoiding the computation time O ( n 2 d ) of using the full matrix multiplication AX . While we view our main contribution to be of a theoretical flavor, on real-world benchmark datasets, we also demonstrate the run-time/memory improvements of our proposed schemes over other baselines via wall-clock time and peak-memory usage comparisons and MSE comparisons with respect to the observation budget.

7 Discussion and Future Work

Our result on learnability with the run-time complexity reduction is particularly significant when the number of nodes n is large and the feature dimension is much smaller compared to n (i.e., d ≪ n ). In addition, we note that the result holds even when the graph adjacency matrix and the data matrix are dense (e.g., weighted graphs where the a ij entries of the adjacency matrix are non-negative real values, and not restricted to { 0 , 1 } ). Examples include pairwise similarity matrices and Erdos-Renyi graphs with constant edge probability p , which have a number of edges that scales as n 2 . In such cases, sparse matrix multiplication would not help much, and our approach could be very useful.

Our proposed scheme is not only useful for speeding up the training of linear GCNs. It is also useful in partial observation settings where the algorithm only has access to a partial view of the graph, and we provide theoretical guarantees that this partial view is still sufficient to accurately perform the regression task. On the contrary, conventional speed-up methods such as sparse matrix multiplication are purely a technique for computational purposes and still require complete information on A and X . For instance, consider a social network setting where, in order to know friendships between users, a learner needs to use an API to query the friendships. Note that one needs fewer queries for using our method (since it requires partial A ) than for using the sparse matrix multiplication (which needs full knowledge of A ). If the graph is large, the query complexity gain becomes larger.

Several extensions of our results are possible. First, establishing theoretical guarantees for the nonlinear GCN framework, and for other target tasks, such as graph link prediction and node classification, would be of significant interest. Based on the empirical observation that our training scheme works well for a two-layer non-linear GCN, extending our theoretical results to the non-linear GCN under some assumptions (as in [34]) on non-linear activation functions seems to be promising. Note that our proposed sampling scheme (based on approximating the leverage score) and the theoretical guarantee we made are tailored for the node-level regression task in the training phase. The reason we focused on this specific setting was so that we could take advantage of the theoretical results on leverage score sampling and linear regression in order to obtain theoretical results for GNN training using a subsampled graph. In particular, we are helped by the fact that in the linear GCN setting, the optimal solution for the linear regression is a matrix-vector product which can be expressed as (( AX ) ⊤ ( AX )) † ( AX ) ⊤ y , where A , X , and y denote the adjacency matrix, data matrix, and the labels respectively.

For graph link prediction and classification tasks, on the other hand, since the optimal solution cannot be expressed as a matrix-vector product (due to non-linearities and different cost functions), our proposed algorithm does not extend to the classification problems in a straightforward way, and similarly, our theoretical guarantee may not hold anymore. For future work, we want to study whether our same subsampling process can have any guarantee for a node classification task. For more general tasks, we believe that we need different sampling strategies other than the leverage score sampling and hence we may need other tools for developing a theoretical guarantee for sampling algorithms.

Another direction for future work is to establish generalization guarantees beyond the approximation errors [35]. Lastly, motivated by the fast acceleration of gradient descent algorithms [24-28] and iterative schemes that sample a subset of the entries of A [21, 36], developing algorithms that adaptively select a subset ( A,X ) to observe based on the information at each gradient descent step would be an interesting direction to explore.

Acknowledgments

We thank Ruosong Wang, Yun Yang for the helpful discussion and the anonymous reviewers for their constructive feedback. Han Zhao and Seiyun Shin were partly supported by the Defense Advanced Research Projects Agency (DARPA) under Cooperative Agreement Number: HR00112320012, an IBM-IL Discovery Accelerator Institute research award, and Amazon AWS Cloud Credit. The work of Ilan Shomorony was supported in part by the National Science Foundation (NSF) under grant CCF-2046991.

References

Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. , volume 2, pages 729-734. IEEE, 2005.

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks , 20(1):61-80, 2008.

Max Welling and Thomas N Kipf. Semi-supervised classification with graph convolutional networks. In J. International Conference on Learning Representations (ICLR 2017) , 2016.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems , 29, 2016.

Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493 , 2015.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Carl Yang, Jieyu Zhang, Haonan Wang, Sha Li, Myungwan Kim, Matt Walker, Yiou Xiao, and Jiawei Han. Relation learning on social networks with multi-modal graph edge variational autoencoders. In Proceedings of the 13th International Conference on Web Search and Data Mining , pages 699-707, 2020.

Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics , 33(14):i190-i198, 2017.

Xiang Yue, Zhen Wang, Jingong Huang, Srinivasan Parthasarathy, Soheil Moosavinasab, Yungui Huang, Simon M Lin, Wen Zhang, Ping Zhang, and Huan Sun. Graph embedding on biomedical networks: methods, applications and evaluations. Bioinformatics , 36(4):1241-1251, 2020.

Zhenqin Wu, Alexandro E Trevino, Eric Wu, Kyle Swanson, Honesty J Kim, H Blaize D'Angio, Ryan Preska, Gregory W Charville, Piero D Dalerba, Ann Marie Egloff, et al. Graph deep learning for the characterization of tumour microenvironments from spatial protein profiles in tissue specimens. Nature Biomedical Engineering , pages 1-14, 2022.

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 974-983, 2018.

Statista. Number of social media users worldwide from 2018 to 2022, with forecasts from 2023 to 2027, 2022. URL https://www.statista.com/statistics/278414/ number-of-worldwide-social-network-users/ .

Steven J Cook, Travis A Jarrell, Christopher A Brittin, Yi Wang, Adam E Bloniarz, Maksim A Yakovlev, Ken CQ Nguyen, Leo T-H Tang, Emily A Bayer, Janet S Duerr, et al. Whole-animal connectomes of both caenorhabditis elegans sexes. Nature , 571(7763):63-71, 2019.

Wenbing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. Adaptive sampling towards fast graph representation learning. Advances in neural information processing systems , 31, 2018.

Jie Chen, Tengfei Ma, and Cao Xiao. Fastgcn: fast learning with graph convolutional networks via importance sampling. arXiv preprint arXiv:1801.10247 , 2018.

Ziqi Liu, Zhengwei Wu, Zhiqiang Zhang, Jun Zhou, Shuang Yang, Le Song, and Yuan Qi. Bandit samplers for training graph neural networks. Advances in Neural Information Processing Systems , 33:6878-6888, 2020.

Kenneth L Clarkson and David P Woodruff. Low-rank approximation and regression in input sparsity time. Journal of the ACM (JACM) , 63(6):1-45, 2017.

Mu Li, Gary L Miller, and Richard Peng. Iterative row sampling. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science , pages 127-136. IEEE, 2013.

Jelani Nelson and Huy L Nguyên. Osnap: Faster numerical linear algebra algorithms via sparser subspace embeddings. In 2013 ieee 54th annual symposium on foundations of computer science , pages 117-126. IEEE, 2013.

Michael B Cohen, Yin Tat Lee, Cameron Musco, Christopher Musco, Richard Peng, and Aaron Sidford. Uniform sampling for matrix approximation. In Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science , pages 181-190, 2015.

Michael B Cohen. Nearly tight oblivious subspace embeddings by trace inequalities. In Proceedings of the twenty-seventh annual ACM-SIAM symposium on Discrete algorithms , pages 278-287. SIAM, 2016.

David P Woodruff et al. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science , 10(1-2):1-157, 2014.

Zeyuan Allen-Zhu, Zheng Qu, Peter Richtárik, and Yang Yuan. Even faster accelerated coordinate descent using non-uniform sampling. In International Conference on Machine Learning , pages 1110-1119. PMLR, 2016.

Yurii Nesterov and Sebastian U Stich. Efficiency of the accelerated coordinate descent method on structured optimization problems. SIAM Journal on Optimization , 27(1):110-123, 2017.

Roy Frostig, Rong Ge, Sham Kakade, and Aaron Sidford. Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization. In International Conference on Machine Learning , pages 2540-2548. PMLR, 2015.

Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. A universal catalyst for first-order optimization. Advances in neural information processing systems , 28, 2015.

Naman Agarwal, Sham Kakade, Rahul Kidambi, Yin Tat Lee, Praneeth Netrapalli, and Aaron Sidford. Leverage score sampling for faster accelerated regression and erm. arXiv preprint arXiv:1711.08426 , 2017.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs, 2020. URL https://ogb.stanford.edu/ .

Jure Leskovec and Julian Mcauley. Learning to discover social circles in ego networks. Advances in neural information processing systems , 25, 2012. URL https://snap.stanford.edu/ index.html .

R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters , 33(3):291-297, 1997.

Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graphsaint: Graph sampling based inductive learning method. arXiv preprint arXiv:1907.04931 , 2019.

Sergei Ivanov and Liudmila Prokhorenkova. Boost then convolve: Gradient boosting meets graph neural networks. arXiv preprint arXiv:2101.08543 , 2021.

Aarshvi Gajjar, Christopher Musco, and Chinmay Hegde. Active learning for single neuron models with lipschitz non-linearities. In International Conference on Artificial Intelligence and Statistics , pages 4101-4113. PMLR, 2023.

Jason D Lee, Ruoqi Shen, Zhao Song, Mengdi Wang, et al. Generalized leverage score sampling for neural networks. Advances in Neural Information Processing Systems , 33:10775-10787, 2020.

Seiyun Shin, Han Zhao, and Ilan Shomorony. Adaptive power method: Eigenvector estimation from sampled data. In International Conference on Algorithmic Learning Theory , pages 1387-1410. PMLR, 2023.

David C Hoaglin and Roy E Welsch. The hat matrix in regression and anova. The American Statistician , 32(1):17-22, 1978.

Samprit Chatterjee and Ali S Hadi. Influential observations, high leverage points, and outliers in linear regression. Statistical science , pages 379-393, 1986.

Petros Drineas, Michael W Mahoney, and Shan Muthukrishnan. Sampling algorithms for l 2 regression and applications. In Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm , pages 1127-1136, 2006.

Daniel A Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. In Proceedings of the fortieth annual ACM symposium on Theory of computing , pages 563-568, 2008.

Christos Boutsidis, Michael W Mahoney, and Petros Drineas. An improved approximation algorithm for the column subset selection problem. In Proceedings of the twentieth annual ACM-SIAM symposium on Discrete algorithms , pages 968-977. SIAM, 2009.

Xin Liu, Mingyu Yan, Lei Deng, Guoqi Li, Xiaochun Ye, and Dongrui Fan. Sampling methods for efficient training of graph convolutional networks: A survey. IEEE/CAA Journal of Automatica Sinica , 9(2):205-234, 2021.

Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. Layerdependent importance sampling for training deep and large graph convolutional networks. Advances in neural information processing systems , 32, 2019.

Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. arXiv preprint arXiv:1710.10568 , 2017.

Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 257-266, 2019.

Koen Jochmans and Martin Weidner. Fixed-effect regressions on network data. Econometrica , 87(5):1543-1560, 2019.

Arun Venkitaraman, Hermina Petric Maretic, Saikat Chatterjee, and Pascal Frossard. Supervised linear regression for graph learning from graph signals. arXiv preprint arXiv:1811.01586 , 2018.

Can M Le and Tianxi Li. Linear regression and its inference on noisy network-linked data. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(5):1851-1885, 2022.

Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015.

A Related Work

Leverage score sampling for linear regression: The history of leverage scores traces back to linear regression problems in statistics [37, 38]. The statistical leverages of a matrix indicate the importance of rows in solving a least-squares problem. It has been well-known that leverage score sampling [39, 40] uses leverage scores to sample a subset of the rows of the matrix and solve the regression task with (1 + ε ) accuracy using only O ( dε -2 log d ) subsampled rows of the data matrix X and in time O ( nd 2 + d 3 ) , and there have been extensive works on reducing the time complexity as well as the sample complexity [41, 18-22]. However, we note that our setting is a graph-weighted linear regression, which involves a matrix multiplication of A and X , taking O ( n 2 d ) time. To the best of our knowledge, our work is the first one that addresses the setting of graph-weighted linear regression and proposes an algorithm with subquadratic time complexity in n .

Sampling on Graph Convolutional Networks (GCNs): Most of the related works focus on reducing the receptive field size via subsampling neighbors of nodes in the input graph. The motivation is that the neighborhood size can grow exponentially with respect to the number of layers due to the message passing operation of each layer. In contrast, we focus on the effect of partial observation in one hidden layer and show that node subsampling can reduce the run time as well as the sample complexity while ensuring the desired performance guarantee. We also highlight that our work provides a theoretical guarantee, whereas many of the cited works below provide empirical results without a theoretical guarantee. For readers interested in works related to sampling on GCNs, we refer to the comprehensive survey in [42]. Here we provide a brief summary.

To mitigate the 'neighbor explosion' problem, most of the related works [6, 16, 15, 43] take layerwise sampling approaches to sample neighbors of the nodes in the previous layer in a top-down manner. Specifically, layer-wise samplers [16, 15, 43] adopt importance sampling to sample neighbors in the lower layer given each node in the upper layer, assuming that the sample size for each layer is independent of each other.

Second, node-wise samplers [6] randomly sample neighbors in the lower layer given each node in the upper layer. In addition, [17] proposed a bandit-based approach to perform node-wise sampling in order to optimize the variance in the vein of layer sampling approaches. See also [44], which proposed a variance-reduced estimator to reduce the receptive field of a single node.

Third, there have been several works [45, 32] that adopt graph-wise sampling approaches. In particular, the proposed schemes first partition the node set of a graph [45] or sample subgraphs [32], and then train models on those partitions or subgraphs in a batch mode [3]. They show that the training time of each epoch may be much faster compared with layer-wise sampling approaches.

Graph-weighted linear regression: We note that there have been a few works that address linear regression on network-linked data [46-48]. Yet neither sample complexity analysis nor time complexity analysis is provided.

B Basic Algorithms

1: Draw I j ∼ Bern( p j ) , ∀ j ∈ [ n ] independently, where p j = c ̂ ℓ j ([ AX | -y ]) ε -2 log n

Algorithm 4 LEVERAGESCORESAMPLING ( A, { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } ) Input: Adjacency matrix A ∈ R n × n , leverage score estimate of AX { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } Output: ˜ A ∈ R O ( dε -2 log n ) × n , scaled labels ˜ y ∈ R O ( dε -2 log n )

2: ˜ A ← ∑ n j =1 I j √ p j e j A j : 3: ˜ y ← ∑ n j =1 I j √ p j y j e j 4: return ( ˜ A, ˜ y )

Algorithm 5 REGRESSIONSOLVER ( ˜ AX, ˜ y )

Input:

Matrix ˜ AX ∈ R O ( dε - 2 log n ) × d , scaled labels ˜ y ∈ R O ( dε - 2 log n )

Output:

Weight vector ˜ w ∈ R d

C Proof of Proposition 2

Define t := max v ( x ⊤ i v ) 2 ∥ X v ∥ 2 2 . Hence, it follows that

Equivalently, this means that t is the smallest value such that

Let w = X v . Since X has full column rank, we know that v = X † w . Without loss of generality, assume ∥ w ∥ 2 = 1 , then we can equivalently transform the above constraint as

However, note that ( X † ) ⊤ x i x ⊤ i X † is a rank-one matrix, hence it suffices to choose t such that

This completes the proof.

D Proof of Lemma 1

In this section, we prove Lemma 1, which yields a good approximation by sampling with probability proportional to the leverage score overestimates:

Lemma 3. Given a data matrix X = ( x ⊤ 1 ; . . . ; x ⊤ n ) ⊤ ∈ R n × d and a vector u = ( u 1 , . . . , u n ) of leverage score overestimates of X (i.e., ℓ i ( X ) ≤ u i , ∀ i ∈ [ n ] ), suppose we create a matrix ˜ X by including the i th row of X in ˜ X with probability p i = min[1 , cu i ε -2 log n ] for some constant c , scaled by 1 / √ p i . With probability at least 1 -n -Ω(1) , for any v , we have

To prove the lemma, we rely on the following matrix Chernoff bound, which is a variant of the matrix Chernoff bound in [21, Lemma 11 ] (also, see [49]).

Suppose W 1 , . . . , W n are independent, random positive semidefinite matrices with size d × d . Define Z := ∑ n j =1 W j . If W j ⪯ R · E [ Z ] , then

where A ⪯ B is equivalent to saying v ⊤ ( A -B ) v ≥ 0 for matrices A and B and for all v . To apply this inequality, define W j := 1 p j x j x ⊤ j , which is sampled with probability p j . We first know that Z is an unbiased estimate of X ⊤ X since

First, consider the case where p j < 1 . This implies that

In order to invoke the matrix Chernoff bound, we only need to verify that W j ( ⪰ 0) is spectrally upper bounded:

where the last inequality holds due to the proof of Proposition 2. Now, the statement follows from the matrix Chernoff bound with W ′ j = ( X ⊤ X ) -1 / 2 W j ( X ⊤ X ) -1 / 2 and R = 1 cε -2 log n , to conclude that

with probability at least 1 -d ( n -c/ 3 + n -c/ 2 ) = 1 -n -Ω(1) , for some c > 0 and the regime where d < n . Note that we use a union-bound technique to derive the desired high probability guarantee.

Now consider the case where p j = 1 , where W j := x j x ⊤ j is sampled with probability 1 . In this case, (14) generally does not hold. However, since sampling W j with probability 1 is equivalent to selecting W (1) j , . . . , W ( cε -2 log n ) j , each equal to 1 cε -2 log n x i x ⊤ with probability 1 , it follows that

Here we assume that cε -2 log n is an integer for simplicity. Replacing Y j with ∑ cε -2 log n i =1 Y ( i ) j yields the desired result since it does not change E [ Z ] . This completes the proof.

E Proof of Theorem 1

We will use the following statement of Matrix Bernstein's inequality [49, Theorem 6 . 1 . 1 ]: Suppose W 1 , . . . , W n are independent, random matrices with common dimension d 1 × d 2 satisfying

Define Z := ∑ n j =1 W j . Then for all t > 0 , where

To apply this inequality, first recall that Algorithm 1 samples A : j X j : to get a rank1 matrix ̂ A : j X j : := I j p j A : j X j : with probability

Defining W j := ̂ A : j X j : -A : j X j : , we note that E [ W j ] = 0 as shown in (6). In addition, assuming that entries of A ∈ R n × n and X ∈ R n × d satisfy | A ij | ≤ M and | X ij | ≤ M , we then have

where ( i ) follows from the fact that we are interested in the regime where B n = o (1) ; and ( ii ) follows from the fact that ∥ A : j ∥ ≤ √ nM and ∥ X j : ∥ ≤ √ dM . We can also bound ∥ E [ Z ⊤ Z ] ∥ as

and E [ ZZ ⊤ ] can be similarly bounded, implying that v ( Z ) ≤ n 3 dM 4 B . Setting t = ε ∥ AX ∥ yields

where ( iii ) follows from Assumption 1 that ∥ AX ∥ ≥ n 3 / 2 d 1 / 2 ; and ( iv ) follows from the setting that we focus on d < n . Hence, as long as we set B = Cε -2 log n for some constant C ( > 3 M 4 > 0) , the exponent of the Matrix Bernstein inequality can be bounded as -C ′ log n for some C ′ > 0 , and thus can bound the deviation probability to be n -Ω(1) .

Now using the reverse triangle inequality,

we can conclude that the following spectral approximation holds

with probability at least 1 -n -Ω(1) . This completes the proof.

F Proof of Lemma 2

We will use the following statement of Bernstein's inequality to get the desired result: If W 1 , . . . , W n are independent zero-mean random variables such that | W i | ≤ M with probability 1 . Then for all t > 0 ,

∣ ∣ To apply this inequality, we first provide the following proposition which will serve as key ingredients for proving the lemma.

Proposition 3. Fix ( i, j ) ∈ [ n ] × [ d ] . Let W k := I k p k · [ A : k X k : ] ij -[ A : k X k : ] ij . Then for all k ∈ [ n ] , the following two hold:

Note that these two hold for all ( i, j ) ∈ [ n ] × [ d ] .

Proof. We first bound | W k | as:

where ( i ) follows from the fact that we are interested in the regime where B n = o (1) . We also bound the second moment of W k as:

where we used the facts that E [ I ( ℓ ) ] = p j . This completes the proof.

ij ■

Now setting t = ε · e ⊤ i AX e j and applying the Bernstein's inequality now yields:

where ( i ) follows from a union-bound technique; and ( ii ) follows from Assumption 2 that e ⊤ i AX e j = Ω( n ) , ∀ ( i, j ) ∈ [ n ] × [ d ] . Setting B = C ε 2 log n for sufficiently large C > 16 3 M 4 > 0 , one can observe that the last bound is upper bounded by n -Ω(1) . Since the other direction holds similarly, we conclude that with probability 1 -n -Ω(1) , with budget of B = C ε 2 log n ,

(1 -ε ) · e ⊤ i [ AX | -y ] e j ≤ e ⊤ i [ ̂ AX | -y ] e j ≤ (1 + ε ) · e ⊤ i [ AX | -y ] e j , ∀ ( i, j ) ∈ [ n ] × [ d ] . Pick v ∈ R d +1 , where it can be represented as

for some α 1 , . . . , α d +1 . From the above, one can readily see that with probability at least 1 -n -Ω(1) ,

̂ Choosing v ∗ := arg max v ( e ⊤ i [ ̂ AX |-y ] v ) 2 ∥ [ ̂ AX |-y ] v ∥ 2 2 and w ∗ := arg max w ( e ⊤ i [ AX |-y ] w ) 2 ∥ [ AX |-y ] w ∥ 2 2 , from the above, we have

we get the desired result of

This completes the proof.

G Theoretical Guarantees for Algorithm 2

In this section, we provide theoretical guarantees for Algorithm 2. As mentioned in Section 4, we obtain the same series of theoretical guarantees as those for Algorithm 1. For completeness, we leave the detailed statements below.

Theorem 3. Under Assumption 1, if we run Algorithm 2 with the budget B = Cε -2 log n for large enough C > 0 , the estimate AX satisfies with probability at least 1 -n -Ω(1) .

Lemma 4. Under Assumption 1, Algorithm 2 with the budget B = Cε -2 log n for large enough C > 0 outputs the leverage score overestimates with a ( 1+ ε 1 -ε ) 4 multiplicative ratio; i.e., with probability at least 1 -n -Ω(1) .

Theorem 4. Under Assumption 1, suppose we run Algorithm 2 with the budget B = Cε -2 log n for large enough C > 0 , to obtain { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } . Subsequently, if we run LEVERAGESCORESAMPLING ( A, { ̂ ℓ i ([ AX | -y ]) , ∀ i ∈ [ n ] } ) , we obtain ˜ A ∈ R O ( dε -2 log n ) × n and ˜ y ∈ R O ( dε -2 log n ) such that REGRESSIONSOLVER ( ˜ AX, ˜ y ) provides

We note that it suffices to provide the proof of Theorem 3, as the rest follows in a similar manner to those for Algorithm 1.

Proof of Theorem 3: As in Appendix E, we will use the Matrix Bernstein's inequality described in (16). Recall that Algorithm 2 samples A : j X j : to get a rank1 matrix ̂ A : j X j : := I j p j A : j X j : with probability as in (11)

such that ∑ j p j ≤ B .

Defining W j := ̂ A : j X j : -A : j X j : and we note that E [ W j ] = 0 as shown in (6). In addition, assuming that entries of A ∈ R n × n and X ∈ R n × d satisfy | A ij | ≤ M and | X ij | ≤ M , we then have

where ( i ) follows from the fact that ∥ A : j ∥ ≤ √ nM and ∥ X j : ∥ ≤ √ dM ; and ( ii ) follows from the fact that we are interested in the regime where B = o ( n ) . We can also bound ∥ E [ Z ⊤ Z ] ∥ as

Also, one can readily see that E [ ZZ ⊤ ] can be similarly bounded, implying that v ( Z ) ≤ n 3 dM 4 B . Setting t = ε ∥ AX ∥ yields:

where ( iii ) follows from Assumption 1 that ∥ AX ∥ ≥ n 3 / 2 d 1 / 2 ; and ( iv ) follows from the setting that we focus on d < n . Hence, as long as we set B = Cε -2 log n for some constant C ( > 3 M 4 > 0) , the exponent of the Matrix Bernstein inequality can be bounded as -C ′ log n for some C ′ > 0 , and thus can bound the deviation probability to be n -Ω(1) .

Now using the reverse triangle inequality, we can conclude that the following spectral approximation holds

with probability at least 1 -n -Ω(1) . This completes the proof.

H Further Implementation Details

In this section, we provide additional details on the explanations of the datasets, wall-clock time with the adoption of sparse matrix multiplication, and peak memory usage comparisons.

Datasets: We consider benchmark datasets from the Open Graph Benchmark (OGB) [29], Stanford Network Analysis Project (SNAP) [30], and House dataset [31]. For the MSE comparisons, we consider (1) ogbl-ddi dataset from OGB, (2) ego-Facebook dataset from SNAP, and (3) House dataset. We note that while the third dataset has its data matrix (i.e., a concatenation of feature vectors) and labels, the first two datasets do not have feature matrices and target variables. Accordingly, to conduct controlled experiments, we synthetically generate data matrix X , weight vector w , and noisy labels

y by setting the linear relationship between labels and features: y := X w + n . Here n denotes an additive Gaussian noise with parameters ( µ, σ ) = (1 , 10) . Based upon the size of the two datasets described below, we consider the data matrices X ∈ R n × d (with n ∈ { 4267 , 4039 } and d = 100 ), where each row (i.e., each node's feature vector) is drawn according to the Cauchy distribution with parameters ( x 0 , γ ) = (10 , 100) , in an i.i.d. manner. Here x 0 denotes the location parameter, indicating the location of the peak of the distribution, and γ denotes the scale parameter, specifying the half-width at half-maximum (HWHM). Also, we consider w ∈ R d generated by the Cauchy distribution with ( x 0 , γ ) = (10 , 100) .

ogbl-ddi dataset [29]: This dataset represents a undirected and unweighted graph which models the drug-drug interactions. Each node corresponds to an FDA-approved or experimental drug. An edge indicates interactions between drugs that can be interpreted as the difference between the joint effect of taking the two drugs together and the expected effect when drugs act independently of each other. As there are neither feature vectors nor labels that are available, we synthetically generate those as mentioned above.

ego-Facebook dataset [30]: This dataset models a social network that represents friendship of 'circles' between users from Facebook. Examples of such relationships between users include students of common universities, sports teams, relatives, etc. Notice that the graph adjacency matrix is more sparse compared to ogbl-ddi dataset. As above, since there exists no feature vectors and labels that are available, we synthetically generate those to perform regression task.

House dataset [31]: This dataset represents a graph in which nodes correspond to the properties; the existence of edges means that two properties are located close to each other; and the target label is the property's price. In addition, there exist node features representing MedInc, HouseAge, AveRooms, AveBedrms, Population, and AveOccup. In particular, we use the refined dataset from [33].

Furthermore, for the wall-clock time and peak memory usage comparisons, we additionally use ogblarxiv [29] dataset, representing the citation network between all Computer Science (CS) arXiv papers (although the dataset is tailored for classification of 40 subject areas) and synthetic dataset where all ( A,X, y ) are generated according to Gaussian distribution with size specified in Table 1, Table 2, and Table 3.

Additional results and analysis:

Wall-clock time with the adoption of sparse matrix multiplication: Notice that as compared to results in Table 1, Table 2 demonstrates that the computation time of both our approach and using exact AX become faster due to the adoption of sparse matrix multiplication and the reduction time was larger for using exact AX (especially for the ogbn-arxiv dataset which is sparse). However, we observe that our proposed scheme is faster than the regression by using the exact multiplication of AX . Furthermore, for the synthetic dataset where A and X are generated according to a Gaussian

distribution and hence A is dense, we observe that the sparse matrix multiplication does not help speed up the computation time and the matrix multiplication with scipy.sparse package took even longer than the matrix multiplication with numpy package.

Peak memory usage: In addition to run-time, we expect our proposed approach to offer significant improvements in terms of memory usage. As a simple experiment in terms of memory usage, we provide the peak memory usage comparison for the same datasets during the sampling and regression steps of the algorithm. As shown in Table 3, in the best case, the proposed algorithm requires 1414× less memory than the algorithm using the exact computation of AX .

Computing architecture for wall-clock time and peak-memory usage comparisons: For a fair comparison on Table 1, Table 2, and Table 3, we use the same regression solver and use the same specification of 48 cores of an x86 64 processor with 503.74GB memory.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work tackles the problem of scaling GNNs used for regression tasks to large graphs by subsampling given nodes. The technique consists of first performing node sub-sampling to estimate the leverage scores of $AX$ and then performs leverage score sampling on $AX$. The authors show that this technique is a good spectral approximation to the full $AX$ computation, but avoids the $O(n^2d)$ matrix multiplication cost.\n\nSTRENGTHS:\nThe paper tackles the interesting and practically useful problem of scaling GNNs to large graphs. The proposed technique has useful theoretical guarantees and as far as I am aware it is a novel approach. The approximation to the regression problem is interesting as it has useful spectral guarantees and could be of general interest from an algorithmic perspective outside of the GNN community.\n\nWEAKNESSES:\nThere are a few weaknesses of the work, some of which the authors point out. The most glaring one is the fact that this work targets only linear GNNs. While analysing non-linear models is clearly more challenging, it would be interesting to have some comments or results for non-linear models with more easy to study non-linearities (such as ReLU). \n\nI found following the different run-times a bit challenging. It may be useful to have a summary table that compares the various computational/memory complexities for the different techniques used in the experiments in Section 5. \n\nAs this problem is a practical one, I believe Section 5 could be stronger. While the authors argue that the matrix multiplication is a bottleneck, modern GPUs are extremely optimized for exactly these sort of operations and as a consequence it may be the case that the full matrix multiplication is still faster than sampling. As such, I would be very interested to see some results on the runtimes of the various techniques. Further, it would be interesting to test regimes in which the graph is too large to fit in a GPU and sampling is absolutely necessary. \n\nIt may also be useful to run experiments with non-linear GNNs using the proposed sampling technique.\n\nI also spotted some typos and formatting issues in the manuscript (line 60, line 95).\n\nQUESTIONS:\nWhile the work mentions linear GNNs, it seems to be the case that the work only considers one-hop linear GNNs. While a multi-layer linear GNN is still a linear one, do these results apply to multi-hop linear GNNs as well? It would be useful to have more details on the number of layers and exact model used in section 5 as well.\n\nIs there a reason in figure 2 (b) and (d) that MSE with uniformly sampled $AX$ is not included in the experiments?\n\nLIMITATIONS:\nThe authors acknowledge a main limitation that is the fact that this work only studies linear GNNs. A further limitation is that while the authors acknowledge the improved theoretical run-times of their technique, they do not give practical wall-clock runtimes for the experiments. While the technique may be theoretically faster, in practice many operations such as matrix operation are extremely optimized.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to overcome the computational difficulties often encountered when using graph neural networks (GNNs) for large-scale datasets, by subsampling in the adjacency and feature matrices. By assuming a two-layer linear GNN for the regression problem and performing subsampling based on the leverage score, the authors show that the MSE of the reduced problem generated by their proposed method is bounded from above by the true MSE times $(1+\\epsilon)$.\n\nSTRENGTHS:\nOriginality & Significance: The proposed method basically follows the common idea in subsampling algorithms for linear regression in a standard setting; the same is true for the derivation of the upper bound of the reduced problem MSE. However, in the current problem, the regression coefficients are multiplied with the product of the adjacency matrix and the feature matrix, and the computation of the product itself is expensive and should be avoided. A new twist is introduced at this point: the proposed algorithm breaks the problem into two steps; in the first step the leverage score is approximately estimated by subsampling the rows of the adjacency matrix and the corresponding columns of the feature matrix; in the second step, the reduced adjacency matrix is generated according to the estimated leverage score. In this way, the direct computation of the product is avoided. I think this idea is interesting. Subsampling algorithms with theoretical guarantees are currently rare in this field, and this makes the proposed method valuable. The originality and significance of the proposed method are considered to be thus high.\n\n\nQuality: The explanation of the algorithm and the derivation of the theoretical guarantee are very clear. The numerical experiment is also convincing. The quality of the paper is thus high.\n\n\nClarity: The presentation is nice and clear, except that the references are missing in the main manuscript (they are in the supplementary material). I think the explanations of the numerical experiment result should be more detailed, though the important parts are understandable. Some typos are also found. Although there are small flaws like these, the overall clarity is high.\n\nList of typos found:\n\nLine 95: discussio -> discussion\n\nLine 186: unnecessary space seems to exist before ``Algorithm''. \n\nLine 225: unnecessary space seems to exist before ``Algorithm''. \n\nLine 226: Appendix Appendix G -> Appendix G \n\nLine 245: 4039 -> 4039 nodes\n\nLine 248: 100),in -> 100), in\n\nWEAKNESSES:\n- The linear GNN is not employed in practice, and the extension to nonlinear GNN is desired. I think it is at least possible to apply the proposed algorithm to the nonlinear case, although there is no theoretical guarantee in that case. Some more discussion or experimental results in such a nonlinear case are welcome.\n\n- The extension to the more-than-two layer case is also nontrivial. Some more discussions are welcome.\n\nQUESTIONS:\n- Concerning to the experimental section, how about the computational time in practice? The proposed method comes from the computational issue and hence it is better to show the computational time/cost as the result. \n\n- Figures 2b and 2d are the magnified views of 2a and 2c, respectively. Is this right? If so, it is better to mention this point.\n\nLIMITATIONS:\nThe authors address their research limitations well. I think there is no concern about the potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

comments: SUMMARY:
The authors proposed a sampling method to train graph neural networks efficiently. However, the current implementation of graph neural networks is based on the sparse matrix multiplication and thus the complexity is not O(n^2d). The authors only analyze the complexity theoretically without experimental supports. Both of major issues make the contribution weak.

STRENGTHS:
1. Theoratically, the proposed method is efficient compared with the dense implementation of graph neural networks.
2. The experiments are conducted on the large scale datasets with different variant of the methods.

WEAKNESSES:
1. The statement about the complexity of the graph neural network is not correct. The current implementation is based on the sparse multiplication, so the complexity is not O(n^2d).
2. The computational time comparision is missing, which is essential for this work.
3. There is lack of baselines, it only compares with its own variants.

QUESTIONS:
1. What is the computational time of this approach compared with sparse matrix multiplication?
2. Besides MSE, how about applying this to the real semi-supervised classification tasks? Will this lead to the same prediction accuracy?
3. Is it possible to use this for link prediction tasks when dropping edges for validation? Is there still any theory guarantee in this case?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an efficient GNN training solution through node subsampling and leverage score sampling, which is proven to be efficient in learning a regression model with bounded entry access and running time.\n\nSTRENGTHS:\n* The proposed technique leads to a proven efficient approach for GNN training, with potentially many real applications. \n\n * Experimental study on real graph data validates the proposed technique on real world graphs, showing significant improvement over baseline sampling strategies.\n\nWEAKNESSES:\n* It is interesting to see the proposed approach for GNN training, as efficiency of GNN on big graphs is a headache for many real world problems. However, although being different approaches, we can find the idea of subsampling used in various GNN models, but it is not well compared in this paper. For example, GraphSage performs neighbor subsampling and GraphSaint subsamples subgraphs from a big graph. The authors should compare the proposed approach with those different sampling approaches to show the pros and cons. \n\n* The experimental results show the significant efficiency of the proposed method. However, since it is for GNN training, the only evaluation on MSE would draw some concerns, if MRS is the only metrics GNN should be concerned about. If not, the authors should expand the experimental study for a comprehensive understanding/evaluation. \n\n* Compared to MSE, it is critical to see how much typical GNN machine learning tasks would be impacted by the subsampling. For example, graph link prediction, node classification, label prediction, etc. This paper does not include evaluation form those perspectives. \n\n* Although the big-O notation of complexity analysis is quite helpful, the experimental study of efficiency should be provided, if the training time is also significantly reduced with comparable performance. \n\n* It is a bit concerned that, for studying GNN efficiency, the authors selected two small graphs from OGB, although much bigger graphs exist.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}