Calibrate and Debias Layer-wise Sampling for Graph Convolutional Networks
Authors: Yifan Chen
Tianning Xu
Dilek Hakkani-Tur
Di Jin
Yun Yang
Ruoqing Zhu
Venue: Accepted by TMLR ()

Abstract: Multiple sampling-based methods have been developed for approximating and accelerating node embedding aggregation in graph convolutional networks (GCNs) training. Among them, a layer-wise approach recursively performs importance sampling to select neighbors jointly for existing nodes in each layer. This paper revisits the approach from a matrix approximation perspective, and identifies two issues in the existing layer-wise sampling methods: suboptimal sampling probabilities and estimation biases induced by sampling without replacement. To address these issues, we accordingly propose two remedies: a new principle for constructing sampling probabilities and an efficient debiasing algorithm. The improvements are demonstrated by extensive analyses of estimation variance and experiments on common benchmarks. Code and algorithm implementations are publicly available at \url{https://github.com/ychen-stat-ml/GCN-layer-wise-sampling}.

Published in Transactions on Machine Learning Research (01/2023)
Calibrate and Debias Layer-wise Sampling for Graph Convolutional Networks
Yifan Chen' Tianning Xul
Dilek Hakkani-Tur
Di Jin Yun Yang'
Ruoqing Zhu'
University of Illinois Urbona- Champeiin "mmazon Aleza Al
{yifanc10, tx8, yy84, rqzhu)ei1linoiss edu
(hakkanit, (yjinamzn)@amazonn com
Reviewed on OpenReview: httpo///ooooott///////o//oo/oo/n
Abstract
Multiple sampling-base baaiin methods have been developed for approximating and accelerating node embodding aggregation graph convolutional networks (GCNs) training. Among them, layer-wise approach recursively performs importance sampling select neighbors jointly for existing nodes in each layer. This paper revisits the approach from matrix approximation perspective, and identifies two issues in the existing layer-wise sampling methods: suboptimal sampling probabilities and estimation biases induced by sampling without replacement. To address these issues, we accordingly propose two remedies: a new principle for constructing sampling probabilities and an efficient debiasing algorithm. The improvements are demonstrated by extensive analyses of estimation variance and experi- ments on common benchmarks. Code and algorithm implementations are publicly available at https://githhii hutp::://gttuub..cmm/cceeem wtw.cchnnsttttt--0----- layen-llarraaaa wis--wwweeeee--ee- sampling..
1 Introduction
Graph Convolutional Networks (Kipf & Welling, 2017) are popular mothods for learning node representations. However, computationally challenging to train a GCN over large-scale graphs due to the inter-dependence of nodes in graph. In mini-batch gradient descent training for an L-layer GCN, the computation of embeddings involves not only the nodes in the batch but also their L-hop neighbors, which is known the phenomenon of "neighbor explosion" (Zeng el al., 2020) "neighbor expansion" (Chen et al., 2018a; Huang et al., 2018). To alleviate such computation issue for large-scale graphs, sampling-basee methods are proposed to accelerate the training and reduce the memory cost. These approaches can be categorized as node-wise sampling approaches (Hamilton et al., 2017: Chen et al., 2018a), subgraph sampling approaches (Zeng et al., 2020; Chiang et al., 2019; Cong et al., 2020), and layer-wise sampling approaches (Chen et al., 2018b; Huang al., 2018; Zou et al., 2019). We focus on layer-wise sampling in this work, which enjoys the efficiency and variance reduction by sampling columns of renormalized Laplacian matrix in each layer.
This paper revisits the existing sampling schemes in layer-wise sampling methods. We identify two potential drawbacks in the common practice of layer-wise sampling, especially on FastGCN (Chen et al., 2018b) and LADIES (Zou et al., 2019). First, the sampling probabilities are suboptimal since a convenient while unguaranteed assumption fails to hold on many common graph benchmarks, such as Reddit (Hamilton et al., 2017) and OGB (Hu al., 2020). Sccondly, the previous implementations of layer-wise sampling methods perform sampling without replacement, which deviates from their theoretical results, and introduce biases in the estimation. Realizing the two issues, accordingly propose the remedies with new principle to construct sampling probabilities and debiasing algorithm, as well as the variance analyses for the two propositions.
* Equal contribution. The majority of this work was done prior the first author's internship at Amazon Alexa AL.

---

Published in Transactions on Machine Learning Research (01/2023)
To the best of our knowledge, our paper is the first to recognize and resolve these two issues, importance sampling assumption and the practical sampling implementation, for layer-wise sampling on GCNs. Specif- ically, we first investigate the distributions of embedding and weight matrices in GCNs and propose a more conservative principle to construct importance sampling probabilities, which leverages the Principle of Max- imum Entropy. Secondly, we recognize the bias induced by sampling without replacement and suggest. a debiasing algorithm supported by theoretical analysis, which closes the gap between theory and practice. We demonstrate the improvement of our sampling method by evaluating both matrix approximation error and the model prediction accuracy on common benchmarks. With our proposed remedies, GCNs consis- tently converge faster in training. We believe our proposed debiasing method can be further adapted to more general machine learning tasks involving importance sampling without replacement, and we discuss the prospective applications in Section 7.
1.1 Background and Related Work
GCN. Graph Convolutional Networks (Kipf & Welling, 2017), as the name suggests, effectively incorporate the technique of convolution filter into the graph domain (Wu el al., 2020; Bronstein el. al., 2017). GCN has achieved great success in learning tasks such as node classification and link prediction, with applications ranging from recommender systems (Ying et al., 2018), traffic prodiction (Cui et al., 2019; Rahimi et al., 2018), and knowledge graphs (Schlichtkrull et al., 2018). Its mechanism will be detailed shortly in Section 2.1.
Sampling-baser GCN Training. To name a few of sampling schemes, GraphSAGE (Hamilton et al., 2017) first introduces the "node-wise" neighbor sampling scheme, where a fixed number of neighbors are uniformly and independently sampled for each node involved, across every layer. To reduce variance in node-wise sampling, VR-GCN (Chen et al., 2018a) applies a control variate based algorithm using historical activation. Instead of sampling for each node separately, "layer-wise" sampling is a more collective approach: the neighbors are jointly sampled for all the existing nodes in each layer. FastGCN (Chen et al., 2018b) first introduces this scheme with importance sampling. AS-GCN (Huang et al., 2018) proposes alterna- tive sampling method which approximates the hidden layer to help estimate the probabilitics in sampling procedures. Then, Zou et al. (2019) propose a layer-dependent importance sampling scheme (LADIES) to further reduce the variance in training, and aim to alleviate the issue of sparse connection (empty rows in the sampled adjacency matrix) FastGCN. In addition, for "subgraph" approaches, ClusterGCN (Chiang et al., 2019) samples a dense subgraph associated with the nodes in a mini batch by graph clustering algorithm; GraphSAINT (Zeng et al., 2020) introduces normalization and variance reduction in subgraph sampling.
To provide more scalable improvement on sampling-baser GCN training, we focus on history-obliiviou biyer-wise sampling methods (e.g., FastGCN and LADIES), which do not rely on history information to construct the sampling probabilities. Note that though some other sampling based methods, such as VR- GCN and AS-GCN, enjoy attractive approximation accuracy by storing and leveraging historical information of model hidden states, they introduce large time and space cost. For example, the training time of AS-GCN can be "even longer than vanilla GCN" (Zeng et al., 2020). Moreover, they cannot perform sampling and training separately due to the dependence of sampling probabilities the training information.
Debiasing algorithms for weighted random sampling. In practice, layer-wise sampling is performed by a sequential procedure named as weighted random sampling (WRS) (Efraimidis & Spirakis, 2006), which realizes "sampling without replacement" while induces bias (analyzed in Scction 5). Similar phenomena have been noticed by some studies on stochastic gradient estimators (Liang et al., 2018; Liu et al., 2019; Kool et al., 2020), which involve WRS as well; some debiasing algorithms are accordingly developed in those works. In Section 5, we discuss the issues of directly applying existing algorithms to layer-wise sampling and propose more timo-cfficient debiasing method.
2 Notations and Preliminaries
We introduce necessary notations and backgrounds of GCNs and layer-wise sampling in this section. Debiasing-related preliminaries are deferred to Section 5.
2

---

Published in Transactions on Machine Learning Research (01/2023)
2.1 Graph Convolutional Networks
The GCN architecture ure for semi-supervised node classification is introduced by Kipf & Welling (2017). Sup- pose we have an undirected graph G = (V,E), where V is the set of n nodes and & is the set of E' edges. Denote node i in V as 04 where i E [n] is the index of nodes in the graph and [n] denotes the set (1. (1.2.....n)) n). Each node Uj E V is associated with a feature vector I2 E I' and a label vector ys E R9, In a transductive setting, although we have access to the feature of every node V and every edge &, i.e. the n x n adjacency matrix A, we can only observe the label of partial nodes Verain C V; predicting the labels of the rest nodes in V - Virain therefore becomes semi-supervised learning task.
A graph convolution layer is defined as:
z(+ - THOWW' w H H =Z(Z))
(1)
where a is an activation function and P is obtained from normalizing the graph adjacency matrix A; HO is the embedding matrix of the graph nodes the 1-th layer, and W is the weight matrix of the same layer. In particular, H is the p feature matrix whose i-th row Tg. For mini-batch gradient descent training, the training loss for an L-layer GCN is defined as Poatch EeyEooen E(y,, 24 ) where is the loss
function, batch nodes Vooth is subset of Virain at each iteration. 2, (L) is the i-th row in 2(4), |-1 denotes the cardinality of a set.
where D is a. diagonal matrix with Dj: = 1 Σ, A.j. The matrix P constructed as a renormalized Laplacian matric to help alleviate overfitting and explod- ing/van /aaviiiiiinn gradients issues (Kipf & Welling, 2017), which previously used by Kipf & Welling (2017); Chen et al. (2018a); Cong et al. (2020).
In this paper, we set p= (1
2.2
Layer-wise Sampling
To address the "neighbor explosion" issue for graph neural networks, sampling methods are integrated into the stochastic training. Motivated by the idea to approximate the matrix PH in (1), FastGCN (Chen et al., 2018b) applics importance-samplinn each node in the I-th layer, they sample set of neighbors 8()) from V with importance sampling probability Pi, where p; x Σ}= PI and Σ P: = 1. For the 11-t layer, they naturally set (1-1) = S(), LADIES (Zou et al., 2019) improves the importance sampling probability as
based strategy. Instead of individually sampling neighbors for
p. x EoJENO P2 Vi Vi e [n]
(2)
- where .VO1 = -Vre and , 1. 1. In this case, s(), the nodes sampled for the 1-th layer, are guaranteed to be within the neighborhood of y() The whole procedure can be concluded by introducing diagonal matrix (())) paxa and row selection matrix Q()) R=*x" which are defined as
(1) 2
1 j=
$0
(sip (())
j= else,
(3)
0, else
0,
where 9t is the sample size in the 1-th layer and {it join are the indices of rows selected in the I- th layer. The forward propagation with layer-wist sampling can thus be equivalently represented as g(41) = (((+1))S() HOW H()) = ( where ((+1) the approximation of the embedding matrix for layer 1.
3 Experimental Setup
History-oblivious layer-wisc sampling methods, the subject of this work, rely on specific assumptions construct the sampling probabilities. A paradox here is that the original LADIES (the model we aim improve on) would automatically be optimal under its own assumptions. However, it is important to verify
3

---

Published in Transactions on Machine Learning Research (01/2023)
their assumptions on common real-world open benchmarks, where major empirical evaluation will be performed (c.f. Section 4). In advance of discussions on existing issues and corresponding remedies in Section 4 and Section 5, we introduce the basic setups of main experiments and datasets across the paper. Details about GCN model training are deferred to the related sections.
Table 1: Summary of datasets. Each undirected edge counted once. Each node in binary labels. "Deg." refers to the average degree of the graph. "Feat" refers to the number of features. training/validation) walp/aaiidaaantteeeeee data.
has 112
"Split Ratio" refers the ratio of training
Dataset
Nodes
Edges
Deg.
Feat.
Classes
Tasks
Split Ratio
Metric
Reddit
232,965
11,606,919
50
602
41
1
66/10/24
Fl-score
ogbn-arxiv
160,343
1,166,243
13.7
128
40
54/18/28
Accuracy
ogbn-proteins 132,534
39,561,252
597.0
8
binary
112
65/16/19
ROC-AUC
ogub-mag
736,389
5,396,336
7.3
128
349
1
85/9/6
Accuracy
ogbn-products
2,449,029
61,859,140
50.5
100
47
1
8/2/90
Accuracy
Benchmarks. The distribution of the input graph will impact the effectiveness of sampling methods to great extent we always construct a graph in an adversarial manner to favor one while deteriorate another. To overcome this issue, we conduct empirical experiments on 5 large real-world datasets to ensure the fair comparison and the representative results. The datasets (see details in Table 1) involve: Reddit (Hamilton et al., 2017), ogbn-arxiv, ogbu-proteins, ogbn-mag, and ogbn-products (Hu et al., 2020). Reddit is a traditional large graph dataset used by Chen al. (2018b); Zou al. (2019); Chen al. (2018a); Cong et al. (2020); Zeng et al. (2020). Ogbn-arxiv, Open Graph Benchmarks (OGB) by Hu et al. (2020). Comparod to traditional datasets, the OGB datascts use have a larger volune (up to the million-node scale) with a more challenging data split (Hu et al., 2020). The metrics in Table 1 follow the choices of recent works and the recommendation by Hu et al. (2020).
bgbn-proteins, ogbn-mag, and ogbn-products are proposed in
Main experiments. To study the influcnce of the aforementioned issues, we evaluate the matrix approxi- mation error (c.f. Section 4.3 and Figure 2) of different methods in one-step propagation. This is an intuitive and useful metric to reflect the performance of the sampling strategy on approximating the original mini- batch training. Since the updates of parameters in the training are not involved in the simple metric above, in Section 6 we further evaluate the prediction accuracy testing sets of both intermediate models during training and final outputs, using the metrics in Table 1.
4 Reconsider Importance Sampling Probabilities in Layer-wise Sampling
The efficiency of layer-wise sampling relies on its importance sampling procedure, which helps approximate node aggregations with much fewer nodes than involved. As expected, the choice of sampling probabilities can siguificantly impact the ultimate prediction accuracy of GCNs, and different sampling paradigms more or less seek to minimize the following variance (for the sake of notational brevity, from now on we omit the superseript (I) when the objects are from the same layer)
E QPSHW - QPHW 12-
(4)
where ||. denotes the Frobenius norm. Under the layer-dependent sampling framework, Zou et al. (2019) show that the optimal sampling probability p; for node i satisfies (see Appendix C.1 for a derivation from a perspective of approximate matrix multiplication)
ps x ||QP!|| (-I((I
([HHW) |l
where for a matrix A, Ap and Abl respectively represent the i-th row/column of matrix A.
4.1 Current Strategies and Their Limitation
The optimal sampling probabilities (5) discussed above are usually unavailable during the mini-bateo gradient descent. training due to circular dependency: to sample the nodes in the f-th layer based the probabilities
4

---

Published in Transactions on Machine Learning Research (01/2023)
in Equation (5), need the hidden embedding H which in turn depends the nodes not yet sampled in the (E - 1)-th layer. In this case, FastGCN (Chen et al., 2018b) and LADIES (Zou et al., 2019) choose to perform layer-wise importance sampling without the information from HW In particular, FastGCN (resp. LADIES) assumes || (HW)) |gll x [|pii]] (resp. [[@P[]]), and sets their sampling probabilities as | (resp. ||@pli (I²), [m).
Pi <
The proportionality assumption above seems sensible considering the computation of the hidden embedding H involves P. However, this assumption unguaranteed because of the changing weight matrix W in training, and no previous work (to our knowledge) scrutinizes whether this assumption generally holds. To study the appropriateness of the core assumption, we conduct a linear regression
y 8o 31
(6)
for each layer separately, where x ranging over (WW 1 lI' is the l norm of certain row in HW and y over ||
the norm of the corresponding column in P.
Table 2: Regression coefficients in Equation (6) for 3-layer GCNs trained with LADIES and full-batch SGD respectively. Negative B1 's are highlighted in boldface.
Method
LADIES
bull-b
Dataset
ogba-arxiv
roddit
oghn-pmttinn
oghn-mag
ogin-arxiv
roddis
ogbn-proteins
ogbn-mang
Ag
3.517 + 0.002
0.01
4.102
0.001
3.687 + 0.001
2,304 + 0.002
20.53 + 0.03
3.942
0.001
3.282 0.001
Layer
8.
-0.54 0.01
8.03 1 0.07
0.488
0.004
-0.391 1. 0.004
-0.15 0.01
15.66
0.17
0.375
0.004
-1.03
0.03
R
0.012
0,012
0.023
0.003
0,001
0,008
0.013
0.026
Ao
6,21
1 0.01
4.41 i 0.01
26.95
0.01
10.67 1 0.01
4.01 1 0.01
27.94 1 0.02
23.46 = 0.01
10.26 1 0.01
Layer
A.
1.59 0.05
-38.18
0.17
L,88 +
1.17
21.07 0.02
-35.09
0.15
-1.12
R
0,028
0,008
0,074
0.001
0,051
0,022
0.068
0.021
An 22.21
10.02
4.72 1 0.01
104.924 1 0.03
29.86 1 0.03
19.7210.02
45.82 1 0.03
174.72 1 0.09
41.98 1 0.02
Layer
A. 1.0020.06
0.10 = 0.03
-137.8
= 0.1
-0.13 = 0.08
2.16=0.07
9.19 = 0.19
-367.2 = 1.1
-29.14 0.05
Ra 0.001
0.001
0.100
0.001
0.001
0.002
0.153
0.100
ogbn-arxiv: layer-1
layer-2
layer-3
75
7.5
40
50
5.0
20
25
2.5
0
[|(HW)ull
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
ogbn-mag: layer-1
layer-2
layer-3
7.5
50
5.0
20
2.5
0
0
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
Column norm P: |lpiill
Figure 1: Regression lines (orange) and scatter plots of ([HW))) | in 3-layer LADIES's different layers on ogbn-arxiv and ogbn-mag.
'This scheme decouples the sampling and the training procedure. We can save the training runtime on GPU by preparing sampling on CPU in advance.
2We take the counterpart assumption [[H x [|QP in LADIES as a randomized version of the one [IH x ||li|]] FasGGNN (and therefore focus on the latter for the regression experiments), since [[( I''s are conceptually independent of the subsequent row selection matrix Q. Supplementary regression experiments on |K [[@P collected in Appendix D.
are
5

---

Published in Transactions on Machine Learning Research (01/2023)
Figure presents regression curves of each layer in LADIES on ogbn-arxiv and ogbn-mag. Table 2 summarizes the regression coefficient So, 31, and R2 of 3-layer GCN trained by LADIES or regular mini-batch SGD without node sampling ("full-batch" in short). The regression results demonstrate that the assumption [[l & ||pli|| || is violated on many real-world datasets from two-fold evidence: negative regression coefficients and small R First, regression coefficients for full-batch SGD and LADIES show similar patterns: negative slope fi's appear across multiple layers and different datasets, such as layer of ogbn- arxiv, layer 2 of ogbn-protains, and both layers and of ogbn-mag. The negative correlation clearly violates the assumption, ([HW)) W || Secondly, the R (see Table 2) in the single variable regression is corr"(x; v), which measures the proportion of y's variance explained by x. A positive Br with small R² can only imply positive but weak correlation between [[]] and || however, a weak correlation cannot imply a proportionality relationship. For example, even though the 31 is positive for each layer in Reddit data, corresponding Rs's are at the scale of 0.01. Consequently, the performance of LADIES on Reddit is not surprisingly inferior our method with new sampling probabilitics (LADIES flat Table 4). Experimental setups are collected in Appendix A.4. Additional regression results for FastGCN and our proposed methods are collected in Appendix D.
4.2 Proposed Sampling Probabilities
The small or even negative correlation between (HW) [4]] and ||P||]] (1Qp!) implics the inappropriateness of the proportionality assumption and the resulting sampling probabilities in BEstGNN/LADIES To address this issue, we instead admit that we have limited prior knowledge of HW under the history-oblivious setting, and follow the Principle of Maximum Entropy to assume a uniform distribution of ([l))) With this belief, we propose the following sampling probabilities:
P: x ||@P||],
Vi E [n].
(7)
Compared LADIES in Equation (2), our proposed sampling probabilities p.'s are more conservative. From matrix approximation perspective,
rewrite the target matrix product QPIHW, and only aim to approximate the known part QPI. It turns out that assuming the uniform distribution of the norms of rows in HW can help improve both the variance of the matrix approxination and the prediction accuracy of GCNs, as empirically shown in Section 4.3 and Section 6.
In addition to the empirical results, we compare the cstimation variance of our probabilitics with LADIES in Lemma C.1 under mild condition 3, Specifically, a common long-tail distribution of HW can justify the strengths of our new probabilities. More discussion and visualization on the distribution of HW are provided in Appendix C.
4.3 Matrix Approximation Error Evaluation
FastGCN
LADIES
LADIES+flat
- LADIES+debiased
-4- LADIES flat debiased
Error (Frob Norm)
Reddit
ogbn-arxiv
ogbn-proteins
ogbn-mag
ogbn- bubnn-products
10a
10a
102
10a
10a
10a
1000
2000
1000
2000
1000
2000
1000
2000
1000
2000
# of sampled nodes
Figure 2: Matrix approximation errors of layer-wise sampling methods. The error curves of LADIES show an abnormal U-shape on ogbn-arxiv and ogbn-mag datasets. "flat" and "debiased" denote our proposed methods in Sections 4 and 5 respectively.
We reiterate that the variance depends underlying distribution the row norms HW, and therefore no sampling probabilities can always have smaller variance than others.

---

Published in Transactions on Machine Learning Research (01/2023)
To further justify proposed sampling probabilities, consider the following 1-layer embedding approx- imation error, which evaluates the propagation approximation to the embedding aggregation of a batch:
1 batch
2 ssamplingl Ur = ||Q PH ( Q batch PSH P
2 where batch and ( 2 certain sampling method, respectively; S is the sampling matrix; Qiatch's 's 0, diagonal entries indicate if node is in the batch. The experiments are repeated 200 times, in which we regencrate the batch nodes (shared by all sampling methods) and the sampling matrix for each method. The batch size is fixed as 512, and the numbers of sampled neighbors ranges over 256, 512, 768, 1024, 1536, and 2048. ( fixed and inherited from the trained model reported in Section
sampling are the embedding at the first layer computed using all available neighbors and
In Figure 2, the result of our proposed sampling probabilities (denoted as "LAIDES+faat" blue solid line) is consistently better than that of the original LADIES method (black dashed line) and of FastGCN (black solid line) on every dataset. The debiasing method in this figure will be discussed shortly in the next section.
5 Debiased Sampling
We first make the clarification that in the derivation of previous layer-wise sampling strategies the neighbor nodes are sampled with replacement, proven to be unbiased approximations of GCN embodding. However, in their actual implementations, sampling is always performed wilhout replacement, which induces biases since the estimator remain the same as sampling with replacement. We illustrate the biases in Figure 2, where the matrix approximation errors on ogbn-arxiv and ogbn-mag datasets (sparse graphs with few average degrees) are U-shape for LADIES. The curve indicates that the errors even increase with the nunber of sub-samples, and the approximation performance heavily deteriorated by the biases.
In the following subscctions, dive into the implementation of FastGCN and LADIES, reveal the origin of the biases, propose new debiasing method for sampling without replacement, and study the statistical properties of the debiased estimators.
Remark. We insist on sampling without replacement because it helps reduce the variance of the estimator. For instance, current node-wise sampling GCNs also sample "without replacement"- apply simple random sampling (SRS, sampling with all-equal probabilities), which guaranteed to shrink the variance by a finite population correction (FPC) factor n= (Lohr, 2019, Section 2.3)
5.1 Weighted Random Sampling (WRS)
The implementation of layer-wise importance sampling (without replacement) follows a sequential proce- dure named as weighted random sampling (WRS) (Efraimidis & Spirakis, 2006, Algorithm D). Given set = [n] representing the indices of items { and the associated sampling probabilities {p.} we sample s samples and denote the sampled indices as Ig for k = 1, 1........ With s sampled indices, Fast- GCN/LADIES use the following importance sampling estimator to approximate the target sum of matrices Σ X, (adapted to the notations in this section)
X /
(8)
1=1
which is a weighted average of X1, 's and induces biases when obtained through sampling without replace- ment. We aim lo preserve the linear form Y. = Σi= Xr, in debiasing, and develop new coefficients Bg. for each X1 to make Y, unbiased; the debiasing algorithm is officially presented Section 5.3.
4 and n denote the sample size and the population size. Tn wise sampling, X, represents B BUG where for brevity QP (HW) is denoted as (C) throughout the analysis.
7

---

Published in Transactions on Machine Learning Research (01/2023)
5.2 Analysis of Bias
To analyze the bias of Equation (8), we introduce the following auxiliary notations. In WRS, given the set Se of k previously sampled indices (0 S & S s - S. == 0), the (A + 1)-th random index 18+1 is sampled from the set V - SS of the rest n indices with probabilities
Pj 00) : Pr, ViEV [m];
(R) P
Pt Eee vk [s-
Sk-
With the notations introduced, we are now able to analyze the effect of applying Equation (8) while the WRS algorithm is performed. The expectation of a certain summand / will be
X X1441
X let1
Plx+a
Fi
=E Eiev- Sy iEV-S, S₁
Σ X
(9)
Pla+1
Plats
Pla+1
where F& is the r-algebra generated by the random indices inside the corresponding set Sk, Ak 0, 1,. (8)
1, and the second equation holds because Picti
is Fi-measuraall. The expectation is in general
Ple+t
unequal to the larget Σi=1 X; for k > O, except for some extreme conditions, such as all-equal p; ps's. The bias
in each summand (except for the first term with k = =) accumulates and results in the biased estimation.
5.3 Debiasing Algorithms
We start with review of existing works on debiasing algorithms for stochastic gradient estimators. Given sequence of random indicos sampled through WRS, there two common genres to assign cocfficients to summands in Equation (8). Both of the two genres relate to the stochaslic sum-and-sumple estimator (Liang
et al., 2018; Liu et al., 2019), which can be derived from Equation (9). Using the fact E Xixt1 ΣiEV-s P:
Pla+1
E Ss Xi], a stochastic sum-and-sample estimator of Σ X; can be immediately constructed
X Xla41 Vk = 0, 8-
M
Σ X; JESc
(10)
(R)
Pfa1
To minimize the variance, Liang et al. (2018); Liu et al. (2019) develop the first genre to focus on the last estimator II, and propose methods to pick the initial - 1 random indices. Kool et al. (2020, Theorem 4) turn to the second genre which utilize Rao- Blackwellization (Casella & Robert, 1996) of II.
In fast training for GCN, both of the two genres are somewhat inefficient from a practitioner's perspec- tive. The first genre works well when Eies,,, Ps is close to 1, otherwise the last term in II, X will bring in large variance and reduce the sample elficiency; for the second genre, the time cost to perform Rao- nho--laackwelizationn (Kool et al., 2020) is extremely high (O(2*) even with approximation by numerical in- tegration) and conflicts with the purpose of fast training. To overcome the issues of the two existing genres, we propose an iterative method to fully utilize each estimator II with acceptable runtime to decide the coefficients for each term in Equation (8).
Piz+1
Denote our final estimator with samples as Y.. Algorithm 1 returns the coefficients Bx 's used in the debiased estimator Y. = Σi=1 B, The main idea is to perform recursive estimation Y1, Y2, ... until Y, and thus update 3 accordingly. To be more specific, we recursively perform the weighted averaging below:
0= Y++1 == - Y + ax+1 II Vk = 0, 0.1,,
where a = and 0g+1 is a constant depending on k. Specifically, Yi II, = XI,/PI, is unbiased and the unbiasodness of Y, can be obtained by induction as cach I1+ is unbiased as well. There can be variant choices of ax+1's. 's. For example, the stochastic sum-and-sample estimator (10) sets all ax+1 's as 0 except for 022 1. In contrast, we intentionally specify 0k+1 (n (m-1)(x+1]> motivated by the preference that if all pi are 1/n, the output coefficients of the algorithm will be all 1/s, the same as the ones in an SRS setting.
oTe proof of its unhiasedness hrief and provided by Kool et al. (2020, Appendix C.I).
8

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm 1: Iterative updates of coefficients to construct the ultimate debiased estimator Y,
Input: probabilities {p:) random indices {Tx+1) generated by WRS with (pr)
Output: a length coefficient vector B
Initialize 9 = 0 E RR, ps = 0 (sum of probabilities);
for k +-+0x 0 to s - do
#k+1 (n-R)(++1)*
9 P|++1] 0*+1
for j 0 to k - 10000 do
B|+1| = (1- - -0x+1) + ax+11
end
ps = ps + 1
end
return 3:
5.4 Effects of Debiasing
We evaluate the debiasing algorithm again by matrix approximation error (see Section 4.3). As shown in Figure 2, our proposed debiasing method can significantly improve the onc-step matrix approximation error all datasets. In particular, by introducing the debiasing algorithm, the U-shape curve of LADIES in Figure 2 no longer exists for debiased LADIES.
We also observe if the new sampling probabilities have already been applied (LADIES + flat in Figure 2), an additional debiasing algorithm (LADIES + flat + debiased) only makes marginal improvement on sparse graphs (ogbn-arxiv and ogbn-mag). The observation implies that the effect of debiasing and new sampling probabilities may have overlaps.
We provide the following conjectures for this phenomenon. First, for the bias introduced by sampling without replacement, it is significant only when the proportion of sampled nodes over all neighbor nodes is large enough. With a fixed batch size while an increasing sample size, sparse graphs generally exhibit a larger bias since they have a larger "sampling proportion" " than dense graphs. Second, our proposed sampling probabilities have a flatter distribution than LADIES, which resembles a uniform distribution and implics a smaller bias (there is no bias in SRS). The phenomenon, therefore, implics an efficient practice to apply the debiasing algorithm: users can decide whether to debias the estimation based on the ratio of the sampling size the batch size and the degrees in the graphs.
In addition to the effect matrix approximation, the evidence that the debiasing algorithm also accelerate the convergence and improve the model prediction accuracy will be provided in Section 6.
5.5 Sampling Time
The iterative updates Algorithm induces additional O(s?) time complexity cost sampling per batch, as in the le-th itcration we need update the coefficients for the first k random indices sampled. We first remark the time complexity is comparable to the one of embedding aggregation in layer-wise training, shown in Appendix B. Moreover, since our layer-wise sampling procedure can be performed independently on CPU, it will not retard the training on GPU 7. Note that this decoupling of sampling and training does not hold for some node-wise or layer-wise sampling methods, such as VR-GCN, which requires up-lo-date embedding information.
The experimental results for sampling time in Table further show that the additional cost of debiasing is acceptable compared to FastGCN and LADIES. For example, comparing "LADIES + debiasing" to LADIES, the sampling time only increases from 11.2 ms to 13.6 ms ogbn-proteins. In contrast, vanilla. node-wise sampling takes 831 ms due to the overhead of row-wisely sampling the sparse Laplacian matrix.
Technicalll the sampling results can prepared in advance of the training the GPU, and therefore claim sampling can be performed independently training. Furthermore, we remark the sequential debiasing procedure is not for GPU.
9

---

Published in Transactions on Machine Learning Research (01/2023)
Table 3: Average sampling time (in milliseconds) per batch for layer-wise methods and the vanilla node wise method. The experiment is conducted on CPU. The batch size set 512 and the sample size set 512/1024 (indicated in the following parentheses). The "f" and "d" in "LADIES+f+d" denotes "flat" and "debiased" respectively. More experimental details collected in Appendix A.3.
FastGCN
LADIES
LADIES
LADIES
LADIES
Node-wise
Reddit (512)
10.6 = 0.9
10.9 + 0.3
10.0 0.2
13.1 = 0.3
13.1 = 0.4
632.5 4.3
Reddit (1024)
10.0 + 0.6
11.8 + 0.4
10.4 0.1
17.1 = 0.4
16.1 = 0.6
637.0 + 4.2
arxiv (512)
4.2 + 0.1
8.3 0.1
7.8
11.7 0.2
11.7 = 0.5
585.2 + 3.6
arxiv (1024)
6.9 0.1
9.7
9.0
17.2 0.3
16.5 = 0.3
585.6 = 3.1
mag (512)
16.8 + 0.1
0.1
24.5
30.0 = 0.03
27.8 = 0.1
1084.3 + 1.7
mag (1024)
18.9 + 0.2
28.6 + 0.2
27.7 0.1
36.0 0.2
34.9 = 0.1
1119 + 2.9
proteins (512)
11.1 + 1
11.2 + 0.3
10 10+:22
13.6 + 0.2
12.6 = 0.2
830.9 + 5.3
proteins (1024)
8.9 + 0.2
12.4 0.1
11.4 : 0.1
18.9 + 0.2
18.0 = 0.3
804.2 + 4.6
products (512)
54.8 + 0.7
83.4
+ 1.3
80.3 + 0.4
83.7 + 0.8
83.5 - 0.6
2795.4 + 4.7
products (1024)
57.1 - 0.5
80.8
+ 0.8
78.7 + 0.7
87.0 = 0.6
85.4 = 0.7
2737.7 + 4.8
5.6 Analysis of Variance
In sampling without replacement, the selected samples are no longer independent, and therefore the classical analysis previous works (c.f. Lemma C.2 in Appendix C) cannot be applied to the variance of WRS- based estimators. To quantify the variance under the WRS setting, leverage common technique in experimental design viewing {3) in JEz, Vk E [s] as random variables. 3(88) denote the coefficients assigned to X;'s when the k-th sample is drawn (if & Sk, 3.(5) == 0). This technique derive the same result as in the previous random indices (Tx's) s) setting, while allow a finer analysis of the variance. We can rewrite the variance in Equation (4) as 8
a
E BSC BCB = Σ Var j.k
Vaa
Σ g B
The variance above is determined by the covariance matrix Cov(B), whose (i, (ith) element is Cov (o , ) We provide the following proposition for the covariance matrix Cov(3) in Algorithm 1:
Theorem 1. For all k E [s], let ps. be the probubility of having Sk as the first k samples, gi be the probability of inder not in the samples, and (k) be the probability of both inder i, not in the k samples.
Define (k) Es. Pss (1 -EEES, Pi), where Esays iterates over all Sye that does not contain Then (ar(a((+1) 2 and Cov(3 ((t-1))) I are recursively given as:
(k) ri 1
(1 - 0 Var( +
o a2+1 (k)
(11)
(1 12 Cov( g
(12)
Furthermore, there ewists sequence {auli=1 only depending on k. n such that for all i, Var(s(M))
<
( 1), () )(0)) < vK [s].
Proof and discussions are collected in Appendix C.4. We remark that due to the fixed weights as = the stochastic sun-and-sample estimator (10), its (co)variance is usually larger than ours, especially when & n (intuitively the second term X,./pl. in Equation (10) will cause large variance).
(e-1)
We let. (C) have n columns (rows), and the jkk)-th element the i-th column (row) denoted Bl( (G(g
10

---

Published in Transactions on Machine Learning Research (01/2023)
6 Experiments
In this section, we empirically evaluate the performance of each method on five node prediction datasets: Reddit, ogbn-arxiv, ogbn-proteins, ogbn-mag, and ogbn-products (c.f. Table 1). We denote "LADIES+Gatt, "LADIES+debiased" and provements from Section 4, Section 5, and from both. We compare our methods to the original GCN with mini-batch stochastic training (denoted by full-batch), two layer-wise sampling methods: FastGCN and LADIES. Apart from that, we also implement several other fast GCN training methods, including Graph- SAGE (Hamilton et al,, 2017) (vanilla node-wise sampling while keep using the GCN architecture), VR-GCN (Chen et al., 2018a); and a subgraph sampling method GraphSATNT (Zeng et al., 2020).
tebiassed respoctively as the variants of LADIES with the im-
In training, we use a 2-layer GCN for each task trained with an ADAM optimizer. (Due to limited com- putational resources, we have to use the shallow GCN since the full- batch method and mode-wise sampling methods require much more GPU memory even when I. = 3.) The number of hidden variables 256 and the batch size 512. For layer-wise sampling methods, we consider settings for node sample size:
1. fixed as 512 (equal to the batch size);
2. an "increasing" setting (denoted with a suffix (2)) that double nodes will be sampled in the next layer.
For node-wise sampling methods (GraphSAGE, VR-GCN), the sample size per node is 2 (denoted with a suffix (2)). For the subgraph sampling method GraphSAINT, the subgraph size is by default equal to the batch size. The experimental results are reported in Table 4, in the form of "mean(=std.)", computed based 5 runs. More details of the settings are deferred to Appendix A.1.
6.1
Model Convergence Trajectory
Full-batch
LADIES
LADIES
LADIES+d
LADIES gobnnproteins
Reddit
ogbn-arxiv arxiv
1.00
0.70
0.80
Metric on dev set
0.90
0.60
0.70
/ n y yommn
0.85
0.55
0.65
0.80
0.50
0.60
0.70
0.40
0.50
20
40
60
80
100
120
0
26
50
75
100
125
150
175
0
50
100
150 200
Index epochs
Index epochs
Index epochs
Figure 3: Metrics (detailed in Table 1) on the validation (dev) sets in each epoch. "P" and "d" refer to the "flat" sampling probability and "debiasing" respectively. The layer-wise sampling methods follow the "incrcasing" setting (denoted with a suffix (2) in Table 4).
We first compare the convergence rates of layer-wise methods. The convergence curves on ogbn-proteins and ogbn-products are shown in Figure (FastGCN is excluded for clearer illustration, due to its outlying curve). Complete results of all methods (including node-wise and subgraph) on all five datasets deferred to Figure 4 in Appendix A.2.
As shown in Figure (and Figure in the appendix), our proposed improvements (LADIES + flat, LADIES + debias, LADIES + flat + debias) exhibit faster convergence rate than LADIES (the solid blue curve). The observation implics both the new sampling probabilities ("flat") and the debiasing algorithm can help accelerate the convergence. Specifically, we note that the effect of debiasing is not as significant choosing proper sampling scheme on some datasets (e.g. Reddit and ogbn-products).
11

---

Published in Transactions on Machine Learning Research (01/2023)
6.2 Prediction Accuracy
Table 4: Metrics (detailed in Table 1) on lesting sets of benchmarks. The best results among layer-wise sampling methods and all methods are both highlighted in boldface. The metrics are in percentage (%). The averaged training time for one epoch is in milliscconds and measured on the GPU.
Metrics
Epoch avg. train. time
Reddit
ogbn-arxiiv
ogbu-mag
ogbn-proteins
ogbn-products
ogbu-arxiv
ogbu-products
Full-batch
93. 83.8110.18
66.39-10.28 25
29.60-10.27
65.71
68.3310.16
65.2 i 3.97
703 + 77.8
Node-wise (2)
02.1310.27
64.31-10.30
29.05110.45
65.7610.18
68.7110.07
22.0 3.64
8.34 1.21
VR-GCN (2)
94.62+0.04
67.49+0.22 0.22
28.99++0.40
67.45+0..2
70.90+0.28
86.8 6.09
88.5 2.51
GraphSAINT
89.4720.83
60.58:10.66 62
24.77=00.88
66.33-20.07
62.77=1.04
23.1 3.26
8,26 = 1.11
FasuGON
24.46:12.30
25.44
7.13+0.48
1..444+1.88
26.98
24.1 5.12
7.43 1.04
LADIES
73.86=0.117
60.95:#0.31
24.79:00448
68 08.28=0.005
52.97=1111
19.3 - 3.25
10.3 1.24
w/ flat
90.04.10.11
62.76 7...0...6
27. 1...01
68 18.06
62.64
16.0 1 2.37
8.03 1.07
w/ debias
86.73=0.36
61.35++0.00
25.74=0.80
68.87=0.09
55.92=0.92
19.1 + 3.45
8.41 1.09
w/ flat & debias
89.34110.40
61.9010..43
27.4110.28
67 17.6411.115
62.5710.22
14.6 2.59
8.11 1.09
FastGCN (2)
60.31110.70
30.23:11.10
5.85110.57
58 80.10.066
31.5810.70
24.9 5.09
8.34 = 1.11
LADIES (2)
88.34+00.1
64.01+0.39
28.59+0..9
68. 08.17+0.10
65.24+0.40
21.0 4.00
11.2 1.35
w/ flat
93.64:10.19
66.5611.84
29.5820.10
68-10:10.07
68.47-0.25
23.1 + 4.04
13.1 1.52
w/ debias
92.75++0.22
65.93+0.277
30.08
69.14+0.15
67. 08-1840.
14.0 + 1.94
8.54 1.09
w/ flat & debias
93.59=00.99
66.22:#0.10
29.88=0.34
67.75=0011
68.49=00066
21.0 = 3.60
8.50 1.15
The prediction accuracy (measured by corresponding metrics) on testing sets of different datasets is reported in Table 4. Our proposed methods, which combine the new sampling probabilitics and the debiasing algo- rithm, are comparable to the full-batch training (no node sampling), showing consistent improvement over existing layer-wise sampling methods, FastGCN and LADIES. On most benchmarks, the prediction per- formance of our methods is better than the vanilla node-wise sampling method (GraphSAGE) and Graph- SAINT.
In addition, we would like to remark on the overlapping effect for the flat sampling probabilities and the debiasing algorithm. In Table 4, the accuracy of LADieS++latt-debias is better than LADIES and LADIES+debias on most benchmarks, but this relative improvement is not remarkable on several bench- marks. This phenomenon is also observed in Figure 2 and Figure 3. The only exception is the ogbn-proteins datasot, where LADIES dataset, LADIES and its variants even oulperform "full-batch" GCN and VR-GCN. We tend to believe GCN's accuracy on ogbn-proteins is mainly impacted by factors other than the sampling variance.
is inferior to LADIES and LADIES-debias However,
this
We further remark on a scemingly strange phenomenon that some efficient GCNs have a higher prediction accuracy than full-batch GCN on several datasets. We speculate the reason that a good approximation can recover the principal components in the original embedding matrix, restrain the noise via the sparse / low-rank structure, and serve implicit regularization. There are similar observations (Sanyal et al., 2018; Chen et al., 2021) in Convolutional Neural Networks (CNN) and Transformers well, that applying low- rank regularizer, such as SVD, to the representation of the intermediate layers can improve the prediction accuracy of models.
Since LADIES improves upon FastGCN, alleviates the sparsity connection issue thereof, and performs con- sistently better than FastGCN on our benchmarks, the variants of FastGCN with our methods (*Fast- GCN+flat", "FastGCN+debias" and "FassCCN+fattdebbiss"" are not the focus of this paper. We report their accuracy results in Appendix A for reference, where consistent improvements of "Fast- GCN+fattdebias" over FastGCN are observed on most benchmarks. We also notice that in some cases, the flat sampling probabilities and the debiasing algorithm bring insignificant improvements or even de- creased accuracy. We conjugate that this is because Gast structural deficiency somewhat. distorts the convergence of GCN, as remarked by Zou et al. (2019). To be more specific, FastGCN does not apply a layer-deenndent. sampling strategy, which lcads to the failure to capture the dynamics of mini-batch SGD and the overly sparse layer-wise connection. When the model does not converge well, the bias and variance of sampling is no longer the dominant factor in model's prediction accuracy.
12

---

Published in Transactions on Machine Learning Research (01/2023)
6.3 Training Time
In the right-most columns of Table 4 9, report the training time for ogbn-arxiv and ogbn-product (complete runtime results for 2-layer and 3-layer GCNs are respectively provided in Tables 5 and 6 in Appendix A.3, along with the detailed timing scttings). The difference between the runtime of our proposed methods and LADIES is not significant, since these layer-wise sampling strategies have the same propagation procedure. As for VR-GCN, a much heavier computational cost required than the layer-wise methods as the expense of involving historical embedding, which helps it achieve the best accuracy on three tasks; in particular, its training time is even comparable to the full-batch method ogbn-arxiv data.
Overall, we comment that based on the experimental results, our improvement in sampling probabilities and the proposed debiasing algorithm lead to better accuracy than the other two classical layer-wise sampling methods, FastGCN, and LADIES, while maintaining roughly the same computational cost.
7 Conclusion and Discussion
In this work, we revisit. the existing layer-wise sampling strategies and propose two improvements. We first show that following the Principle Maximum Entropy, a conservative choice of sampling probabilities outperforms the existing ones, whose proportionality assumption on embedding norms is in general unguar- anteed. We further propose an efficient debiasing algorithm for layer-wise importance sampling through iterative updates of coefficients for columns sampled, and provide statistical analysis. The empirical exper- iments show methods achieve high accuracy close to the SOTA node-wise sampling method, VR-GCN, while significantly saving runtime on GCN training like other history-oblivious layer-wise sampling methods.
We remark our debiased importance sampling strategy can be extended to a broader class of graph neural networks, such as node-wist sampling for GCN. Current node-wise sampling methods, e.g. GraphSAGE and VR-GCN, uniformly sample neighbors of each node withou replacement. To further improve the approxi- mation accuracy, node-wise sampling can also introduce importance sampling with our debiasing algorithm. Moreover, our debiasing algorithm can be applied to general machine learning involving sampling among finite number of elements. In addition to the batch sampling for stochastic gradient descent (SGD) train- ing discussed in Section 5.3, the proposed debiasing method can also contribute to sampling-based efficient attention, fast kernel ridge regression, etc.
Acknowledgments
We appreciate all the valuable feedback from the anonymous reviewers and the TMLR editors. Y. Yang's rescarch was supported in part by U.S. NSF grant 2MS-2210717. R. Zhu's rescarch was supported in part by U.S. NSF grant DMS-2210657
9The least. average training time boldfaced, since the training time on GPU sensitive to the hardware and has relatively large standard deviation the best result. cannot significantly outperform the second-best one.
13

---

Published in Transactions on Machine Learning Research (01/2023)
References
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep lcarning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
George Casella and Christian P Robert. Rao-blackwellisstionn of sampling schemes. Biometrika, 83(1):81 94, 1996.
Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. In Jemnifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, 1CML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 Proceedings of Machine Learning Research, pp. 941-949. PMLR, 2018a. URL http:/ http://proocedingg. mlr. w.r.ppees//vee /rees//v0/caa chwn18p.hrt....ccmcmmmmmm html.
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: Fast learning with graph convolutional networks via impor- tance sampling. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenRevvew.nee, 2018b. URL https://openreviiie
Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. Skyformer: Remodel self-attention with gaussian kernel and nystr\" om method. Advances in Neural Information Processing Systems, 34:2122 14-2222-113, 2021.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pp. 257-266. ACM, 2019. doi: (0.1145/32925003388 1145/3292500 3330925.
URL https://doi.or//11
Weilin Cong, Rana Forsati, Mahmut T. Kandemir, and Mehrdad Mahdavi. Minimal variance sampling with provable guarantees for fast training of graph neural networks. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (eds.), KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp. 1393 1403. ACM, 2020. URL https://dl acm. wwwp///g////oooo/////////oooottt 1145/3394466 3403192.
Zhiyong Cui, Kristian Henrickson, Ruimin Ke, and Yinhai Wang. Traffic graph convolutional recurrent neural network: A deep learning framework for network-scale traffic learning and forecasting, 1EEE Transactions 2019.
on Intelligent Trunsportation Systems, 21 (1111):4883-488
Petros Drineas, Ravi Kannan, and Michael W Mahoney. Fast monte carlo algorithms for matrices i: Ap- proximating matrix multiplication. SIAM Journal on Computing, 36(1):132-157, 2006.
Pavlos Efraimidis and Paul G Spirakis. Weighted random sampling with a reservoir. Information Processing Letters, (7(5):181-188 2006.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1024-1034, 2017. URL nttps://proceedeie www.igg...u.rss..e. httppssccccapp///////aaaaa Bddaa5503aaadddttaaa377atteeee www.://ebbbbbbbbaaaat.aaaa
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Newral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL pttps://proceeddnng neurips. cc/paper/ 2020 2000hhaa///EO001116//////err -bsttrratbbbtttt...oooo html.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Wen-bing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. Adaptive sampling towards fast graph rep- resentation learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Biss-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: An- nual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 4563 4572, 2018. URL https:/ https://proceedings wwt.seneerrrrrcceoooooo ctterrppscccpppp//211//////// h0sh//hh//hhh/h//////
01ee000ee160G6111999990066680 Abstrrcccbbbttttt.......ccaa html.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:/ http:::///ppeeeerrre/e/ wwtpeerrrie..ennt/ouutiii..ee.. www.://mnf/ormdd--- SJU4ayYg1.
Wouter Kool, Herke van Hoof, and Max Welling. Estimating gradients for discrete random variables by sampling without replacement. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
1d=rk1E/2EFVB.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V. Le, and Ni Lao. Memory augmented pol- icy optimization for program synthesis and semantic parsing. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in New- val Information Processing Systems 31: Annual Conference on Newral Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 10015-10027, 2018. URL https:// wwt.:est-.bsttttt....... html.
Runjing Liu, Jeffrey Regier, Nilesh Tripuraneni, Michael I. Jordan, and Jon D. McAuliffe. Rao- blacwlaaceelliizedd stochastic gradients for discrete distributions. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.); Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 4023 4031. PMLR, 2019. URL http: http:///proceeding.. mlr war..peee press/v97/liul9/ html.
Sharon L Lohr. Sampling: design and analysis. Chapman and Hall/CRC, 2019.
Afshin Rahimi, Trevor Cohn, and Timothy Baldwin. Semi-supervised user geolocation via graph convolu- tional networks. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguis- tics (Volume 1: Long Papers), pp. 2009-2019, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10. (5.18853/v1 1 URL https: attps://aclanthhooogy wrw.p/g....r.///......
Amartya Sanyal, Varun Kanade, Philip HS Torr, and Puneet K Dokania. Robustness via deep low-rank representations. arXio preprint Xxx::1804.0709) 2018. URL httts:://arti www.aaaw...rrrr////o./eee.eeeeee org/abs//800 07090.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling, Modeling relational data with graph convolutional networks. In European semantic web conference, pp. 593-607. Springer, 2018.
Zonghan WI, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprchensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 2020.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Yike Guo and Faisal Farooq (eds.). Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pp. 974 983. ACM, 2018. doi: 10.1145//211919932199999 URL /ttps:///oi... 10. 1145/3219819 1221999.........9.... 3219890.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graph- saint: Graph sampling based inductive learning method. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
https://openrew// htt.:a/wweee.oorr//ddddeeeeeeaae
15

---

Published in Transactions on Machine Learning Research (01/2023)
Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. Layer-dependent importance sampling for training deep and large graph convolutional networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Newral Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancoweer, BC, Canada, pp. 11247-11256, 2019. URL https:// httpo/////aa//aa//a//eee// 2aa9///aahh/9//// proceedings neurips. http://eecaappeeooo/aaaaaaaaaaaa to Abstaaaa Abstractt..ttreee html.
16

---

Published in Transactions on Machine Learning Research (01/2023)
A Supplementary Experimental Setup and Results
A.1 Experimental Setup Details
We describe the additional details of experiment setups for Section 6. All the models are implemented by PyTorch. We use one Tesla V100 SXM2 16GB GPU with 10 CPU threads to train all the models listed in Section 6. Our implementation of full-batch method, FastGCN, and LADIES are adapted from the codes by Zou et al. (2019); the implemcntation of vanilla node-wisc sampling, VR-GCN, GraphSAINT is adapted from the codes by Cong et al. (2020). For the vanilla node-wise sampling method, there are several variants of GNN structures (Ying al., 2018) while we fix the model structure as GCN in our experiments for fair comparison. We use ELU the activation function in the convolutional layer for all the models: ELU(x) = x for > 0, ELU(z)= exp(z) - 1 for 0.
For the details in model training, the learning rate is 0.001 and the dropout rate is 0.2, which means 20 percents of hidden units are randomly dropped during the training. Validation and testing are performed with full-batch inference (our experiments on those graph benchmarks are transductive: all the possible neighbors are accessible during training) on validation and testing nodes. Note that some existing PyTorch implementations of GCNs involve several ad-hoc tricks, such as row-normalizing sampled Laplacian matrix.
For the prediction accuracy evaluation experiments in Section 6, we stop training when the validation Fl score does not increase for 200 batches. For a fair comparison, we remove certain tricks in our experiments, such as normalization of each row in the sampled Laplacian matrix in layer-wise sampling, Such a trick may help in the practice, but it might not be compatible with some other methods and is out of the focus of our study. We use the metrics in Table 1 to evaluate the accuracy of each method. Concretely, Reddit is a multi-class classification task, and we use the Micro-F1 score with function "skicarn.metries.. scooe". For OGB data, we use the built-in evaluator function in module ogb provided by Hu et al. (2020).
A.2 Model Convergence
Figure 4 is a supplementary to Figure 3. We present the convergence curve of all methods on every task. The setting of each model is the same as in Figure 3.
A.3 Sampling Time and Training Time
We compare the sampling time per batch for 1-layer GCN with layer-wise sampling methods (FastGCN, LADIES, and our proposed methods) and GraphSAGE by experiments on CPU. The time is presented in milliseconds, The batch size is 512, and the number of sampled nodes is 512 or 1024. The average sampling time (followed by standard deviation) over 200 batches is presented in Table 3. We note that the sampling time may involve some overhead costs. For example, the input Laplacian matrix is Scipy-spare-matrix on the CPU, while in sampling, it is converted to syyTorrh-pparse-marri.
By Table 3, conclude that the cost of debiasing algorithm is acceptable. Moreover, since the debiasing only depends on the number of nodes sampled, its time cost. will be dwarfed by sampling on very large graphs. For example, sampling 512 nodes, the average batch sampling time for "LADIES", "LADIES + "ebiased", "LADIES + flat + debiased", are 8.3 = 0.1, 1.,, 11.7 + 0.5 respectively on ogbn-arxiv while 83.4 1.3 and 83.7 0.8 and $3.5+0.6 0.6 respectively on ogbn- products data. As we mentioned in Section 5.3, the node-wise sampling takes significantly longer time because individually sampling from each row in the re-normalized Laplacian matrix (stored as a sparse matrix) leads to a large overhead cost.
We present the complete training time (per batch) of 2-layer and 3-layer GCNs in Tables 5 and 6 respectively. The time is presented in millisecond and averaged over 110 batches, where we discard the first 20 and the last 20 out of 150 total batches to disregard potential warm-up time for GPU. The other settings are kept the same as our experiments of accuracy evaluation in Table 4. We note that the timing on GPU is sensitive to the hardware and has a relatively large standard deviation.
As presented in Tables 5 and 6, our proposed methods have similar training time with LADIES due to the same propagation scheme of GCN with layer-wise sampling strategy. VR-GCN generally shows superiority
17

---

Published in Transactions on Machine Learning Research (01/2023)
Full-batch
LADIES LADIES
LADIES+d LADIES+f+d
Node-wise
GraphSAINT
FastGCN
VR-GCN
Reddit
ogbn-arxiv
ogbn- Sgbn-proteins
0.7
Metric on dev set
0.9
0.6
0.7
0.8
0.5
0.6
0.7
0.4
0.6
0.3
0.5
0.5
0.2
0.4
0.1
25
50
75
100
125
0
50
100
150
50
100
150
200
ogbn-producto
ogbn-mag
0.8
0.25
Metric on dev
0.20
0.15
0.10
0.05
0.2
0.00
50
100
150
25
125
Index of epochs
Index of epochs
Figure Metrics of cach epoch on the validation set. The layer-wise sampling methods follow the "in- creasing' setting (denoted with sulfix (2) in Table 4); for the node-wise sampling methods, the number of neighbors is 2 per node.
18

---

Published in Transactions on Machine Learning Research (01/2023)
in prediction accuracy (see Table 4). However, it also takes significantly longer time in training since its propagation involves using and updating historical activation.
Table 5: Average training time (in milliseconds) per batch for 2-layer GCN.
Reddit
ogbn-arxiv
ogbn-mag
ogbn-proteins
ogbn-products
Full-batch
372 + 21.5
65.2 + 3.97
72.3 + 8.25
1702 + 67.0
703 = 77.8
Node-wise (2)
8.13 = 1.12
22.0 3.64
17.3 + 4.67
9.50 1 1.29
8.34 1.21
Node-wise (10)
11.3 = 1.05
19.7 2.84
22.0 i 5.60
10.3 + 1.29
11.7 1.31
VR-GCN (2)
153 + 17.3
86.8 + 6.09
106 + 12.4
239 + 42.7
88.5 + 2.51
VR-GCN (10)
302 + 23.9
104 8.47
175 15.1
360 + 45.6
402 65.3
GraphSAINT
8.23 = 1.14
23.1 + 3.26
20.4 + 5.98
8.34 + 1.07
8.26 = 1.11
FastGCN
9.47 1.21
5.12
23.3 6.27
8.50 + 1.22
7.43 1.04
LADIES
7.95 = 1.03
19.3 3.25
16.7 + 4.54
8.69 + 1.11
10.3 1.24
w/ flat
7.86 = 1.04
16.0 2.37
26.1 + 6.54
9.00 = 1.21
8.03 = 1.07
w/ debiased
7.86 + 1.11
19.1 3.45
19.5 1 5.67
8.21 1 1.04
8.44 1.09
w/ flat & debiased
8.01 + 1.09
14.6 2.59
20.5 1 5.62
9.06 1 1.14
8.11 1.09
FastGCN (2)
9.47 1.25
24.9 5.09
5.71
8.76 + 1.06
8.34 1.11
LADIES (2)
14.2 = 4.68
21.0 = 4.00
22.7 + 6.05
8.83 + 1.12
11.2 - 1.35
w/ flat
8.70 = 1.13
4.04
21.9 5.65
10.3 + 1.24
13.1 1.52
w/ debiased
8.75 = 1.15
14.0 1.94
16.3 + 4.64
10.7 + 1.29
8.51 + 1.09
w/ flat & debiased
8.12 = 1.13
21.0 3.60
13.3 4.15
12.5 + 1.41
8.50 1.15
Table 6: Average training time (in milliseconds) per batch for 3-layer GCN.
Reddit
ogbn-arxiv
ogbn-mag
ogbn-proteins
ogbn-products
Full-batch
1042.8 30.1
148 + 5.8
352.1
11.5
3312.3
82.1
4490.7 102.6
Node-wise (2)
10.9 + 1.3
27 = 4.2
17.2 + 4.3
11.5 = 1.3
11.5 = 1.1
Node-wise (10)
77.8 12
34.9 +: 3.3
24.6
50.4
VR-GCN (2)
379.7 + 23.6
154.1 + 12
218.7 + 16.7
128.9 + 53
473.6 + 69.7
VR-GCN (10)
858.1 32
224.9 17.8
17.8
488.1 = 34.7
1618.7 67.3
2075 + 112.8
GraphSAINT
9.5 + 1.1
22.6 2
21.2 = 5.6
11.2 1.3
10.5 = 1.0
FastGCN
16.9 + 6.3
40.8...33
19 + 4.8
13 1811 1.3
1.0
LADIES
10.6 + 1.3
3.7
19.9 4.8
11.1 +
8.9
w/ flat
10.1
3.9
12.122
10.1 1.2
9.9
w/ debiased
10.1
30 4.1
2.5
10.6 =
10.6 1.0
w/ flat & debiased
9.7 1.1
27.1
15.7 4.3
10.9 + 1.2
9.3 1.0
FastGCN (2)
9.6 9.6
2.64
18.9 4.7
9.8 + 1.2
9.4 1.1
LADIES (2)
10 1.2
4.1
19.2 - 5.1
10.5 + 1.2
10.2 - 1.1
w/ flat
16.1 1 4.9
27.2 1 3.7
21.4 1.8 5.8
10.4 14411
13.1 1 1.4
debinsed
10.3 1 1.1
24.7 3.1
23.3 1 5.8
10.5 1 1.2
12.4 1 1.3
W flat & debiased
10.7 + 1.1
26.8 + 4.4
24.8 + 6.4
10.5 + 1.1
9.9 + 1.1
A.4 Regression Experimental Setup
For the regression experiments in Section 4.1, we train a 3-layer GCN with LADIES sampler or full-batch sampler, with 256 hidden variables per layer. The batch size is 512. Early stopping training policy is applied. The regression lines in Figure 1 are fitted based all the (lI |all [P]|]]] pairs collected from converged 3-layer GCN models in 5 repeated experiments. To make the pattern of the scatter plot clear, we only randomly sample 1000 pairs of data in each layer in each scatter plot. We also remark that these experiments are conducted to check the assumption of importance sampling, rather than pursuing SOTA performance. When we finish training the model, the norms of rows in H W are extracted through full-batch inference with all training nodes.
We collect additional regression experiments in Appendix D.
19

---

Published in Transactions on Machine Learning Research (01/2023)
Testing metrics (detailed in Table 1) for FastGCN with flat sampling or/and debiasing on bench- marks. The metrics are in pereentage (%).
Table 7:
Accuracy Metrics
Reddit
ogbn-arxiv
ogbn-mag mag
ogbn-proteins
ogbn-products
LADIES
73.86 + 0.17
60.95 + 0.31
24.79 + 0.48
68.28 + 0.05
52.97 + 1.11
FastGCN
44.46 + 2.30
25.44 0.82
7.13 = 0.48
52.44 + 1.88
26.98 = 0.42
FastGCN w/ flat
53.78 + 0.86
22.76 + 0.69
5.69 + 0.65
61.68 + 0.50
32.72 + 1.6
FastGCN w/ debias
44.27 + 2.26
26.57 = 2.52
4.06 + 1.17
53.64 + 1.33
26.77 = 1.32
FastGCN w/ flat debias
53.88 0.94
24.56 + 2.1
7.54 + 0.31
60.30 1.89
29.79 2.19
LADIES (2)
88.31 0.11
64.01 + 0.39
28.59 + 0.39
68.17 0.10
65.21 + 0.40
FastGCN (2)
60.31 + 0.70
30.23 + 1.10
5.85 + 0.57
58.80 + 1.06
31.58 1 0.70
FastGCN (2) w/ flat
59.60 1.07
26.68 + 1.96
6.68 = 0.03
64.04 + 0.63
35.27 - 1.14
FastGCN (2) w/ debias
51.11 + 1.93
25.65 + 1.02
5.83 = 0.76
56.30 2.49
27.38 - 1.82
FastGCN (2) w/ flat & debias
60.22 + 0.84
24.41 + 1.02
6.06 + 0.62
63.13 + 0.57
34.47 = 1.76
A.5 Supplementary Accuracy Results for FastGCN with flat sampling probabilities and debiasing
To complement the accuracy evaluation results Table 4, we accordingly implement the variants of FastGCN ("FastGCN w/ flat", "FastGCN w/ debias", and "FastGCN w/ flat & debias") and perform the same experiments (all the five datasets in Section 6.2) them. We still follow the same settings (including all the hyper-parameters in training) described at the beginning of Section 6: "FastGCN (2)" similarly refers to the "increasing" setting, where "double nodes will be sampled in the next layer". We summarize their accuracy results in Table 7, where we also provide the accuracy of original FastGCN and LADIES for comparison.
By introducing flat sampling probabilities and debasing algorithms, we observe "FastGCN w/ flat debias" makes consistent improvements over FastGGNN on most benchmarks. However, some cases, the flat sam- pling probabilities and the debiasing algorithms bring insignificant improvements even decreased accuracy, We provide possible explanation for the phenomenon as follows. As illustrated in Table 7, FastGCN and its variants have significantly worse performance than even vanilla LADIES all the benchmarks, which implies that the poor model convergence of FastGCN is mainly caused by a structural deficiency, the sparse layer-wise connection (Zou et al., 2019), on these benchmarks; regarding the impact on the model accuracy, the structural deficiency far outweighs the variance in sampling, and our improvement on sampling variance cannot always lead to higher model accuracy.
B Time Complexity Analysis
Table 8: The time complexity for computation for I-layer GCN training by layer-wise sampling and node-wise sampling. The first column refers to the matrix operation type, nodes aggregation or linear transformation.
Layer-wise ((scp)
Node-wise
Nodes Aggregation
O(sb"p)
Lincar Transformation
(sp²L)
O(sbi-1p2 1p2)
We analyze the complexity of vanilla node-wise sampling and layer-wise sampling in this section. The analysis is adapted from the work by Zou et al. (2019), but we show a lighter bound for layer-wise sampling. For l such that 0 < 1 < L - 1, the propagation formulas for sampling based GCN can be formulated as:
(0+1= = p 1 (
where 0 = z0) E Raxp, p()) E Ra-Ixe, Wi E RPXP. In particular, for LADIES, P(0) pS())
PLADIES
=
For simplicity, we suppose that the hidden dimension in each layer is fixed as p, the same as the dimension of (0). The batch size and the numbers of nodes sampled in each layer set all equal to fixed constant.
20

---

Published in Transactions on Machine Learning Research (01/2023)
We assume the number of sampled neighbors per node in node-wise sampling is b. We denote the maximal degree of all the nodes in the graph as c. The computational cost of the propagation comes from two parts: the linear transformation, a dense matrix product, (0 and the node aggregation, a sparse matrix product, ( (")). The time complexity is summarized in Table 8. We additionally comment although the time cost of two parts both linearly depend on the number of nodes involved (number of non-zero elements Q((+1),, the node aggregation part usually dominates since the sparse matrix product involved less efficient than the dense matrix product involved in a modern computer.
The linear transformation, (0()) is dense matrix production. The cost depends the shape of two matrices, and is given as (sspp)) LADIES fixes as for each layer, so ((sip²) Osp²)) For node-wise sampling = sb2-1, since the number of node grows exponentially. Thus, by summation over all the layers, we have the results in the second row in Table 8.
The node aggregation, p()) (W() is sparse matrix production, since p(() is sparse. For simplicity, we denote (()))) as C(I) R=xp, Thus, the time complexity of this spanse matrix production becomes O(nnz("))p) where nnz is the number of non-zero entries in p For layer-wise sampling, since we sample nodes for each layer and each node has at most neighbors, sampling, since each node has b neighbors and the neighbors are not shared by all the nodes in each layer, (ny = bst sbb+1-1. By summation over all the layers, we altain the results in the first row of Table 8.
mnz I For nose-wise
C Theoretical Analysis of Sampling Probabilities
C.1 Results in approximate matrix multiplication
In this section, we revisit approximate matrix multiplication to derive the previous layer-wise sampling methods. Specifically, the sampling matrix S used in FastGCN and LADIES can be decomposed as S III"" where II E Raxd is sub-sampling sketching matrix defined as follows:
Definition C.1 (Sub-sampling sketching matrix). Consider a discrete distribution which draws with prob- ability P. > 0, Vi E [n]. For a random matrir II E Raxd, if II has i.i.d. columns and each column IT can randomly be Tap: Cs with probability ps, where e, is the i-th column of the n-by-n identity matrie Ia, then is called a sub-sampling sketching matric with sub-sampling probabilities {p:
With this definition, we introduce a result in AMM to construct the sub-sampling sketching matrix, which coincides with the conclusion in FastGCN and LADIES.
Theorem C.1 (Theorem 1 (Drineas et al., 2006)). Suppose B E R=ax" C E Raxne, the number of sub- sampled columns d E Z such that 1 < d < n, and the sub-sampling probabilities {pr)-- are such that E Pi 1 and such that for a quality coefficient E [0,
B
P 2
8
Vi E [n]. I1Mlll
(13)
EV=1
BB
Construct a sub-sampling sketching matriz II E Raxd with sub-sampling probabilities {pr as in Defini- tion C.1, and let BHIII C be an approzimation to BC. Let ó E (0,1) and = 1 with probability al least 6,
Then
|BC - BUI
<
B BEIC. 3d
(14)
The theorem is closely related to Lemma 1 in Appendix B of LADIES, which studies the variance E BI - BSC For the choice of sub-sampling probabilities, Equation (13) reproduces the conclusion in FastGCN and LADIES, when we respectively take B as and QP.
Remark.
C.2 Comparison of Sampling Variance Between Our Sampling Probabilities and LADIES
Whether our choice of probabilities can outperform the LADIES depends on the distribution of the norms of rows in HW. When () is not proportional to the corresponding Ez norm of column (QP)(), our
21

---

Published in Transactions on Machine Learning Research (01/2023)
proposed probabilities can benefit the approximate matrix multiplication task more than the ones assuming relation of proportionality. We find the common long-tail distribution of numbers suffices to exert the strengths of the new probabilities, which can be concluded as the following assumption:
Assumption 1. To simplify the notation, we denote B == QP and C == HW, where P is an n n-by-n matric
as defined above. Let m be the number of non-zero columns B, and define C1 == (Σ. || /m)
2
There also erists constant C2 21 such that c / S || - C2l|Cilp//m. Assume C,/C3 21
With the assumption above, we show the variance of the approximation with our proposed probabilities is smaller than the variance of LADIES by the following lemma.
Lemma C.1. We denote the sampling matria with our probabilities in Equation (7) as S1, and denote the sampling matria with probabilities of LADIES in Equation (2) as So. If Assumption holds, then we have
E BS,C - BCC E BS. - -CB
Remark. Assumption is related to the uniformity in the distributions of |B and ||aa]|'s. We tentatively discuss the implication of the assumption in Appendix C.2. We remark the assumption indicates it is unrealistic that the new probabilities can outperform the ones in LADIES, as distributions of datasets can vary. Nevertheless, as shown in Section it can be an effective attempt to improve the prediction accuracy LADIES by simply adopting the conservative sampling scheme.
To prove Lemma C.1, we first adapt technical lemma (Zou et al., 2019, Lemma 1), which relates the sampling matrix to the variance (expectation of squared Frobenius norm) of the approximate matrix multi- plication.
Lemma C.2 (Adapted from Lemma 1 (Zou et al., 2019)). Given two matrices B ERaa and C E Raxne, for any i E [n] define the positive probabilities p 's such that Li= 1. We further require the probability Ps = 0 if and only if the corresponding column B or row Cμ is all-zero. The sub sub-sumbpp sampling sketching matria II E gaxd is generated accordingly. Let S = IIII" it holds that
Es |BBCC- -
Σ kp:>0
B
|[
1|
BC
Pi
where d is the number of samples.
With the lemma above, the proof of Lemma C.1 is provided as follows.
Proof. Recall the notation in the main paper is simplified == == QP,C == HW. As the union of
neighbors of nodes in Q cannot cover all the nodes, some columns in B are all-zero, and we accordingly define a Q-mcasurable matrix L as in Lemma C.2. We have
E BS ] = Eq [.., (IBS QC
- R
Σ Pp.
B
2
|C
Pj
where the second equation holds as apply Lemma C.2 to the inner expectation in the right-hand side of the first line. Plugging ps x |B|| (Equation (7) in the main paper) into the preceding probabilities p, 's, we reach
EQ [E
Expraa
B
||
EQ BC
E B BC
]
1
EQ
Bbil]
B
||Gial|
1
EQ ||
d
22

---

Published in Transactions on Machine Learning Research (01/2023)
As computed by Zou et al. (2019), the variance of LADIES is similarly given
-
B [ -|IBS BC
Ee
Σ
Σ Pepi
|
11
Ee [[IBC]]]]
Consequently, to prove the lemma it suffices to show that
1)
< f:p
Σ
B
Σ
B
Σ Exp,
Bld
(15)
and the inequality above follows with Assumption 1. / [n] in the left-hand-side above, we have
Specifically, plugging the inequality
||C|a]l
2 C2 | B
Σ PPP
||
Σ kpp:00
|
Σ
C2 m||C Yum
<
m
|BIl2a/m m
in which the last equation comes from the definition C := <,
To close the proof, we utilize the
/m))
inequality J. /11l//// s || 2|2 and bound m||C|| by nO2 Eip, Ga Finally we attain Equation (15)
with the core assumption
G C 2 1.
Remark. In Assumption we indeed implicitly assume |B Il's follow long-tail distribution that most norms are around the average while a few columns have large norms. The high non-uniformity makes the average of squared norms much larger than the square of averaged norms. For ||C]]]'s,, considering the normalization techniques (such as batch or layer normalization) to stabilize the scale of the parameters, they lend to not vary widely, which implies a small C2. The numerical experiments on the comparison of approximation error (see Figure 2) and the histograms of the norms in trained models shown Figure 5 further validate the assumption. Based on the empirical analysis above, claim the assumption mild and tends to hold at least for some datasets.
C.3 Distributions of the Matrix Rows / Columns Norm
Figure 5 demonstrates the distribution of [Pll|] and (HW)) (layer 1, 2, 3) for Reddit, ogbn-arxiv, ogbn- proteins, and ogbn-mag datasets. The ([(HW) gll' are obtained from the experiment in Section A.4. The outlicrs larger than the 99.9% quantile or small than the 0.1% quantile are removed.
As shown the histograms, our analysis regarding Assumption 1 tends to hold generally on these datasets. For the norms columns in P (as a replacement for QP for clarity), we observe there are some columns with large norms far beyond the average. Those columns contribute a lot the quadratic mean, which results in huge C in Assumption 1. In contrast, the norms of rows in HW concentrate around their average, inducing small C2. Those facts together with Assumption and Lemma C.1 explain why proposed sampling probabilities are more proper for some real datascts.
C.4 Proof of Theorem 1
Proof. We first show E ((((()) 1, Vi E [n], E [s]. As 2!(h is constructed by Algorithm 1 to attain the unbiased estimator, take X; = X, = 0, Vj + i, and we have E 3 EE= ((t) X, = >" 1
[n], k [s).
With E ((++1) at hand, we still need to compute E (a! (and E 8 (1++)) g(++))) 1((+1))((t)) to obtain the (co)variance. To start the analysis, we recursively write g(* (++1) as
Ejes, Pj
B ((++)) (ies.) (8/40 ax+1) ax41] + 1 (igs.) (1eS411
@++1 == m (a(1()( +%
(k+1)
pi
23

---

Published in Transactions on Machine Learning Research (01/2023)
reddit
Relative Frequency
10
0.05
0.04
0.020
8
0.04
0.015
0.02
0.02
2-
0.01
0.01
0.005
0.00
0.00
00.000
0.50
0.75
1.00
60
20
80
20
100
Norm
Norm of rows HW: lay-1
Norm rows in HW: lay-2
Norm rows
lay-3
ogbn-arxiv
Relative Frequency
0.8
0.20
0.06
2.0
0.15
1.5
0.4
0.10
1.0
0.5
0.2
0.05
0.0
0.0
0.00
0.00
Cols P
Norm of
lay-1
in HW: lay-2
Norm of lay-3
ogbn- ogbn-proteins
20
Relative Frequency
0.06
0.008
15.
0.04
0.006
10
2
0.004
0.02
0.00
0.000
Norm Cols P
Norm of HW: lay-1
in HW: lay-2
Norm of lay-3
ogbn-mag
Relative Frequency
0.15
0.04
0.4
0.10
0.02
0.05
0.01
0.0
0.00
0.00
Norm Cols P
Norm of HW: lay-1
Norm in HW: lay-2
Norm of lay-3
Figure 5: Distributions of p |I's and || (HW)
( ll's for Reddit, ogbn-arxiv, ogbn-protein and ogbn-mag.
24

---

Published in Transactions on Machine Learning Research (01/2023)
For E (3 we notice the cross term 2n; 2 first terms, utilizing the fact [(IESs) {( have
(k+1) ( is always zero as (1ES))) (igSz) {
as for the
++ (o(t))) (B(44))((1 axt1)² + 2ax+1 -ax+1) + a
to obtain the last term,
1 1 Σ JEs Pj
E E |² = EEE [iggst ( (iES
On 113 $ Sk
Pi
EjEs, Pj
E
1 ((gSe) E ((IESaNt))
ak+ili & Sk
Pi
1 EjEs, Pj
r
psa
pt ac+1
S 2
For (
o g(ttt), we can similarly drop the last term Ey ( (ett)) as [ssesa) = O; as for the first
term E E (B,(a) ) (1f4)),, we have
E Ex, an ) -- (B)0))
E E
000 ggth ax=12++)2 + (1) + ((EES
) - ak+1) + pr
where phg the probability that both index i,j are the first samples; as for the next term E n, + (=(tt(((t))))))))))))))))) (o(t))) (1+2), we first compute
1 EyEss py
E E (k+1) =EE
7j
8
GES (JESL-1)
00+11 (FR
Pj
1 Exes, Pjr
E
(j
ax+1 E (11sESCH1)
Pj
1 1 E-EYss Ps
8((0)
Pj
(1895))889)) autt.
E
a.+1
Eyes, Pj
Pj
and similarly we have
1 1 Eves, Ps
E Eliiss,17) (++1)
(x+1))
1 (ieSki ( (1eSet1) EE
ax+1 Fr
Pj
EyEes Psr
E
((ES)) ( (1e8s)
-x+1 E (1(1586
Pj
--EeS, Pj'
E ((eSh)) (jg8s)
agt1 1- Eyes, Pr
Pj
E (1jjss)) ((eSs) )ax
Pj
accordingly we can obtain
+ ( k+1 ) (k+1) = E (I aatt(11 (otes,),t))oxt111111 (1++((1-4+1))
E ET
and applying the same derivation above have
E ++1 (3())) (k+1) = - (1( a002(11 - ax-1) + EE(111111 (1es)) (
Combining all the pieces together, we obtain
EE 3((+1) g(h =1, )(() (1 ax+1)² 2 20x+ ax+1) + (pr,) + E (1pes, (ies) + 1 ) ai =E g(x) (1 a2+1)2 ) 20 - a1++1) + (pi, + E (Lues)) (UEs.) + Lugs ( (JES.))) aff1.
25

---

Published in Transactions on Machine Learning Research (01/2023)
With the derivation above, have
(k)
B B(o(a+t)e== =1 - (22+1)²+ + 20x+1 (1 ax+1) +
+ 9: k
Jak+1
pi
( g(++)))(())))))))))) g(++1)= = E ( 0/00 (1 ax+1)² 200++1 (1a4++1) + a
where (= -1th) ) is the probability that index is in the first k samples, and similarly dh of T PL.)) is the probability that either index i or index j is in the first k samples. Plugging the expression above into the following identitios,
/.
Var(B((t1)) E (8) (1( B² (3((t1))
Cov(B glhtt)) E 3 (1+ g(a+1) - EE ((t+11) B g(tt1)
we can then have the expression for the covariance stated in the main paper.
For the scale of the covariance, we prove the upper bound through induction. We can verify the upper bounds hold for & = 1, and for the (co)variance with ag = 1 k Var(B(*+1) and Cov(s .09 g((tt)) now respectively becomes
Var(o((t1))
12
Varls
Fi
(k)
(k)
=
(k + 1)2
+
D1
qi
(k: + 1)2
Cov(s((t1), , g(att))== = ka
Cor(B! ,000)
(k)
Th
(k + 1)²
(k + 1)2
Utilizing the induction conditions that for all i,j, we have
Var( S
1 1);
p;
(ov(( ((()() s k
Along with the facts that
P: q
(k)
(k)
q:
(k)
< 1
1 and
9
(k)
s 1, can finally achieve the inequality
Pi
Pi
that
Var(B(kt11)
1
1 1),
-
k Pr
(Cov(3(
,.g
(((t+t))]
s
k +1
Remark. The choice of ag &/ here is mainly for easing the proof, while may not be the optimal choice in practice; indeed in SRS the og 's are different than the ones used here.
D Supplementary Regression Results
D.1 Full Batch Training
In this subsection, we present the regression analysis for GCN with full-batch SGD training (without sam- pling). Figure shows similar pattern, as supplementary to Figure 1. Here, the scatter plots make use of all points. The assumption: [| ||Pli]]| still does hold. Note that we do not have the regression result the ogbn-product dataset, since the training of 3-layer GCN fails due to memory limitation.
26

---

Published in Transactions on Machine Learning Research (01/2023)
reddit: layer-1
layer-2
layer-3
100
100
100
50
50
50
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
ogbn-arxiv: layer-1
layer-2
layer-3
40
75
50
20
25
2
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
ogbn-proteins: layer-1
layer-2
layer-3
400
5
200
20
4
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
ogbn-mag: layer-1
layer-2
layer-3
8
40
100
4
50
2
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
Column norm of P: ||pli||
Figure Regression of ([HWW) ~ 9o + Pill on Reddit, ogbn-arxiv, ogbn-proteins, and ogbn-mag
datasets. 3-layer GCN trained by full-bacth sampler. The fitted regression line in orange color.
27

---

Published in Transactions on Machine Learning Research (01/2023)
D.2 FatCCN//asSGCN--dbiisin
In this subsection, we present the regression analysis for FasCCCCy/FattCCC--eeiaaang in this subsec- tion. The regression results illustrated in Figures and 8; more details can be found in Table 9. The distribution patterns of the (Il(H)) ll [PP pairs in FastCCN/FastGCN CNNee/aaaGCNNNeeiisstt patterns in Figures 6 and 9; we can analogously draw the conclusion that for models trained by Fast- GCN/FastGCN <ebbasiin, the assumption || (a)) x ||Pli]| tends to not hold.
similar to the
reddit: layer-1
layer-2
layer-3
60
60
40
40
40
20
0
1.0
0.5
0.5
ogbn-arxite layer-1
layer-2
layer-3
10
7.5
5.0
2.5
||1#W)il
0.2
0.4
0.5
0.2
0.4
0.6
0.2
0.4
0.6
ogbn-proteins: layer-1
leyer-2
layer-3
4.0
10
3.5
10-
3.0
0-
0.0
0.2
ogbn layer:1
layer-2
layer:3
3.0
2.5
2.0
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
narm ||pb|]
80 + 81|| on Roddit, ogbn-arxiv, ogbn-proteins, and ogbn-mag datasets. 3-layer GCN trained by FastGCN sampler. The fitted regression line is in orange color.
Figure Regression of (I (gll
Table 9: Regression coefficients for ( all ~ 30 + B | The data come from 3-layer GCNs trained with FastGCN F d(bbiasing) respectively. No regression has high R² and the H² for positive B are highlighted in boldface.
Layer 1
Layer 2
Layer 3
Method
Dataset
8p
81
R2
So
31
R2
3o
B1
R2
ogbu-arxiv
2.651
-0.165
0.005
1.627
0.981
0.017
1.640
0.123
<0.001
reddit
11.773
8.453
0.009
3.639
5.155
0.006
2.524
1.587
0.001
FastGCN
ogbn-proteins
2.853
0.457
0.022
4.692
-5.883
0.069
12.938
-22.552
0.098
ogbn-mag
2.278
-0.057
0.001
1.208
0.023
<0.001
0.938
-0.108
0.005
ogbn-arxiv
2.817
-0.164
0.004
1.984
1.277
0.021
2.385
0.111
<0.001
reddit
13.747
9.712
0.008
4.654
6.322
0.007
3.264
1.563
0.001
FastGCN+d
oooneiroteins
3.090
0.397
0.017
5.372
-7.357
0.072
11.395
-0.179
0.102
ogbu-mag
2.310
-0.067
0.002
1.194
-0.009
<0.001
0.999
-0.147
0.006
28

---

Published in Transactions on Machine Learning Research (01/2023)
reddit: layer-1
layer-2
layer-3
100
40
40
50
20
20
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
ogbn-arxivv layer-1
layer-2
layer-3
7.5
10
5.0
0.2
0.4
0.6
0.2
0.6
ogbn-proteiis: layer-1
layer-2
layer-3
15
10
3.5
10
3.0
0.0
0.2
0.4
0.6
0.0
0.4
0.6
0.0
0.2
0.4
0.6
ogbn-mag mag: layer-1
layer 2
layer-3 3
2
2.0
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
Column norm P: [|Pb|]
Figure 8: Regression of WI 80 + 3 Reddit, ogbn-arxiv, ogbn-proteins, and ogbn-mag
datasots. 3-layer GCN is trained by FastGCN+dcbiastte sampler. The fitted regression line is in orange color.
29

---

Published in Transactions on Machine Learning Research (01/2023)
D.3 LADIES: setting 1
This subsection presents regression plots for ( ~ ||Pl||] as supplementary to Figure Note that different regression setting (HHW)) |QP is collected in Appendix D.4.
reddit: layer-1
layer-2
layer-3
20
40
20
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
ogbn- anxiv: layer-1
layer-2
layer-3
75
6
4
20
25
2
||0=Woull
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
ogbn-proteins: layer-1
layer-2
layer-3
5.0
100
4.5
50
4.0
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
ogbn-mag: layer-1
layer-2
layer-3
8
40
50
4
20
25
2
0.4
0.6
0.2
0.4
0.6
0.2
0.6
Column norm of P: ||pii||
80 + Pill on Reddit, ogbn-arxiv, ogbn-proteins, and ogbn-mag datasets. 3-layer GCN trained by LADIES. The fitted regression line in orange color.
Figure 9: Regression of ([
D.4 LADIES: setting 2
[|QP!(] in LADIES as randomized version of the one [[HWW) |4 x [[P[[]]] and hence focus on the latter regression sctting. However, it may still be interesting to present. regression analysis of || (HW)) [ in this subsection to study the corresponding assumption [[(HW) [|QPit]]| in LADIES. Compared to the original assumption in FastGCN, sctting [Qlll as the predictor causes some different patterns in the regression.
As remarked in the footnote in Section 4.1, view the assumption [I |al]
There are more empty columns QP than P. For a single selection matrix, we have fewer (æ.y) pairs. To compensate, we pick 500 Q's and record non-zero |QPli||s and corresponding ([HW)) the regression input.
30

---

Published in Transactions on Machine Learning Research (01/2023)
There is also higher portion of high-leverage points after considering the selection matrix Q the points with large |QP I's are fewer while they have higher influence the coefficients (which is not favored in regression analysis since they increase the standard error of the estimated coefficients).
The regression results are illustrated in Figures 10 and 11; more details can be found in Table 10. We note the regression results still fail to support the proportionality assumption in LADIES: most 8i 's are negalive, and even for the positive 31 the R² (the coefficient of determination, equal to the square of the correlation coefficient in univariate linear regression) is small, which matches the observation in the figures that there is no clear proportionality in the data.
reddit: layer-1
layer-2
layer-3
40
20
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
ogbn arkiv: layer 1
layer
layer 3
200
100
100
50
2
0.6
0.6
110000011
ogbn-proteins: layer-1
layer-2
layer-3
60
100
40
20
0
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
ogbn-mag: layer-1
layer-2
layer-3
10
100
150
100
50
0.0
0.2
0.6
0.0
0.2
0.4
0.0
0.2
0.6
Column norm of P: ||Pi!||
80 + 8 | on Reddit, ogbn-arxiv, ogbu-proteins and ogbn-mag datasots. 3-layer GCN is trained by LADIES sampler. The fitted regression line is in orange color.
Figure 10: Regression of [I ((
31

---

Published in Transactions on Machine Learning Research (01/2023)
reddit: layer-1
layer-2
layer-3
150
100
200
100
100
50
0.0
0.5
0.0
1.0
0.0
0.5
1.0
ogbn arxiv: layer 1
layer-2 2
layer 3
75
100
5.0
50
2.5
25
50
0.0
0.6
0.0
0.2
0.4
0.6
(00000111
ogbn-proteins: layer-1
layer-2
layer-3
60
150
100
2
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
ogbn-mag: layer-1
layer-2
layer-3
100
0.6
0.4 Column norm P: [|Pi!|]
0.6
Figure 11: Regression of [I (HW)) |g|l ~ 8o + 8 ||Q@!! on Reddit, ogbn-arxiv, ogbn-proteins, and ogbn-mag
datasots. 3-layer GCN is trained by LADIES + debiasiig sampler. The fitted regression line is in orange color.
Table 10: Regression coefficients for ([HH)) ~ 3o + Bi B|llQiill| The data come from 3-layer GCNs trained with 31 are highlighted in boldface.
respectively. No regression has high R² and the R² for positive
Layer
Layer
Layer 3
Method
Dataset
81
R²
3
R2
Sp
B1
R²
ogbn-arxiv
3.819
-0.424
0.007
14.827
-20.833
0.048
32.079
-2,8803
0.028
reddit
12.218
24.459
0.006
6.409
7.711
0.001
5.897
-8.366
0.002
LADIES
ogbn-proteins
4.425
0.642
0.006
33.042
-139.554
0.033
113.204
-405.534
0.060
ogbu-mna
4.138
-0.797
0.011
20.193
-17.247
0.039
48.205
-19.879
0.011
ogbu-arxiv
3.593
-0.538
0.011
6.503
4.125
0.025
19.388
1.374
0.001
reddit
22.122
13.941
0.008
21.783
13.231
0.004
37.739
1.320
<0.001
LADIES +d
ogbu-proteins
4.025
0.296
0.012
24.889
-36.718
0.0S0
109.424
-197.488
0.152
ogbu-maa
4.184
-0.653
0.009
12.524
-0.920
0.001
31.837
0.118
<0.001
32


Reviews:
[{"review_id": "nKEDC6owxu", "reviewer": "Reviewer_9G1C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe authors provide in-depth analysis on the two drawbacks in layer-wise sampling and the results are convincing.\n2.\tThe proposed sampling probabilities (Eq (7)) is improved based on the violated assumpation in FastGCN/LADIES, which in spirit is interesting.  \n3.\tIt seems that the proposed debiasing method can be further adapted to more general machine learning tasks involving importance sampling without replacement, which may broden the application areas.\n\nWeaknesses&Questions:\nThough the overall paper is interesting, I have the following concerns:\n1.\tFrom Table 4, when choosing LADIES(2) as the baseline, the training time increased when additionally applying “flat” on the “debias” variant (21 vs 14). However, when choosing LADIES as the baseline, the training time decreases from “w/ debias”(19) to “w/ flat&debias”(14). Could you provide some explanations?\n2.\tIn terms of some metrics (e.g., ogbn-arxiv), the performance decreases when additionally applying “flat” on the “debias” variant. From these point of view, the proposed two strategies may be conflict in some application sceneries.. \n3.\tFrom Figure2, the “flat&debiased” baseline has similar performance compared with the “debiased” baseline when the number of sampled nodes is relatively small (around 1000). Could the two strategies be automatically selected via some threasholds to achieve a bettern trade-off between effectiveness and efficiency. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors points out two potential drawbacks (sub-optimal sampling probabilities and sampling without replacement) in the common practice for layer-wise sampling of graph convolutional networks. To address these, thr authors 1) propose a more conservative principle to construct importance sampling probabilities relying on the Principle of Maximum Entropy; 2) suggest a debiasing algorithm to deal with the bias induced by sampling without replacement.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please see the weakness part above.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3pH1TS3vRN", "reviewer": "Reviewer_LYct", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper identifies an important theory-practice gap regarding assumptions and biases in layer-wise sampling on GCNs.\n- The proposed solutions are simple and address the identified issues.\n- Experiments include relevant large-scale benchmarks and validate the efficacy of the proposed solutions.\n\n**Weaknesses**\n- I am not sure if this contribution advances the state-of-the-art in (large-scale) node classification since vanilla GCNs are weak models, usually outperformed by simple fast approaches [e.g.,1, 2].\n- The experimental setup only considers shallow models (up to 3 layers).\n- Except for Table 3, it is unclear how time is measured (e.g., if it was measured on GPU or CPU). Reporting only CPU numbers could be misleading.\n\n[1] https://arxiv.org/abs/2010.13993 \n\n[2] https://arxiv.org/abs/1902.07153", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates two issues on layer-wise sampling schemes for GCNs (FastGCN and LADIES). The first one refers to wrong assumptions on sampling probabilities, which is tackled by adopting the Maximum Entropy principle. The second issue refers to biased estimation due to sampling without replacement, for which the paper introduces a debiasing sequential procedure. Experiments on relevant benchmarks show better convergence and accuracy of LADIES combined with the proposed solutions.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall, this is a solid and interesting work. I only have a few minor requests/comments:\n\n- Adding to the captions whether time was measured on GPU or CPU (like in Table 3) would be helpful. Also, when the paper says \"our procedure can be performed independently on CPU, it will not retard the training on GPU\", it neglects CPU-GPU communication time. Is this communication time insignificant here? Somehow I would expect the sequential debiasing procedure on GPU to cause a higher effect on the overall training time. \n- It would be interesting to report accuracy numbers for FastGCN+flat+debias in the Appendix.\n- Saying that the success of GCNs comes from the \"successful approximation to the spectral graph convolutions\" (Sec. 1.1) is questionable. For instance, I would simply begin with \"GCN has achieved great success ...\".\n- The paper would benefit from a textual review. Here are some typos:\n  - Sparese (Page 2)\n  - Matrix \\hat{D} should be bold. (Page 2)\n  - Use \\mathcal{S} (before equation 2)\n  - We insist ON sampling ... (Remark)\n", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TGjR9MTUiQ", "reviewer": "Reviewer_31rs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengthens:\n\n* This paper points out two problems related to the traditional sampling strategy used in layer-wise sampling for GCN; the problems might be interesting to the GCN research community.\n* The proposed methods are simple and obtained encouraging performance as reported in the paper.\n\nWeaknesses:\n\n* Some motivational analyses and statements are not convincing.\n* More empirical evidence should be given to show the generality of the identified problems.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to improve layer-wise sampling for graph convolutional networks (GCN), with a particular focus on two popular models, i.e., FastGCN and LADIES. The authors pointed out two shortcomings of the current sampling methods: 1) the sampling probabilities are suboptimal because they are often constructed under unguaranteed assumptions; 2) the sampling is performed without replacement, bringing in unexpected estimation biases. Accordingly, the authors proposed two methods to address these issues respectively. They propose to generate sampling probabilities following the Principle of Maximum Entropy and introduce an iterative debiasing algorithm. Results show that the proposed methods could reduce the matrix approximation error and improve downstream performance.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\n\n\n1) The analysis for the sampling probability problem in Section 4 is not convincing\n\n    a)\tFirst of all, it is FastGCN that assumes $||(\\mathbf{H}\\mathbf{W})_{\\[i\\]}||\\propto ||\\mathbf{P}^{\\[i\\]}||$ rather than LADIES. Why do the authors examine the correlation between them for LADIES? For LADIES, it should be $||\\mathbf{Q}\\mathbf{P}^{\\[i\\]}||$. Such a mismatch makes the results in Table 2 and Figure 1 less convincing, so it’s still questionable whether the traditional assumption really deteriorates the sampling probability.\n\n    b)\tSecondly, stating that “little information of $||(\\mathbf{H}\\mathbf{W})_{\\[i\\]}||$ can be explicitly retrieved from $\\mathbf{P}$” is annoying. Both position and negative coefficients could tell that these two variables are correlated, albeit with a different relationship. Instead, by using your proposed approach dropping the approximated part or assuming a constant of 1 for $||\\mathbf{P}^{\\[i\\]}||$, the correlation is totally ignored. \n\n    c)\tThirdly, from intuition, the proposed sampling method should perform better in settings where negative coefficients are observed but worse in those positive-coefficient setups. But this is not true. In Table 4, the “flat” method performs consistently better than the LADIES baseline on Reddit (where LADIES show positive coefficients across all layers). Could you explain this observation? This may be related to the mismatch problem in a).\n\n    d)\tThe debiasing algorithm introduced in Section 5 shows promising performance in Table 4. Would fixing the bias problem also alleviates the probability issue? What about the coefficients for “LADIES/FastGCN + debiasing”?\n\n2) The identified problems are assumed to be general to layer-wise sampling based GCN, but the authors only performed a full examination with LADIES. Since the authors also focused on FastGCN in this paper, it makes a lot of sense to provide full results for FastGCN as well, which could offer great insights to the readers on whether the findings in this paper are generalizable. \n", "broader_impact_concerns": "I didn't find serious ethical problems.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to improve efficiency of graph neural networks. The paper focuses on improving sampling strategy for graph convolution networks and in this regards, the author begin by identifying shortcomings on current methods like LADIES or FastGCN, like bias caused by sampling without replacement. A new sampling distribution with flattening and a debiased sampling algorithm is proposed. Strong empirical improvements is achieved across several large-scale benchmarks when correcting LADIES approach, but unfortunately the proposed method seems to not work well with FastGCN. Nevertheless, the proposed method and technique are correct, experimental result demonstrate effectiveness in one way, and the approach will be of interest to the community, hence I propose to accept the paper with some minor modifications.

1. Add limitation about proposed method not working for FastGCN. It can't be just sparsity of FastGCN is the cause as adding the debiased sampling almost always hurts FastGCN's performance.
2. As pointed out by 31rs, it would be instructive for readers to write about why proposed method with flattening still works better than LADIES on datasets where proportionality assumption holds? Is it better hyper-parameter tuning?
3. Rename appendix D.3, it can't be titled author response.

I will thank the authors for the patience and all the updates for more clarity and thorough evaluation.
Decision: Accept (poster)