Title: Layer-Neighbor Sampling --- Defusing Neighborhood Explosion in GNNs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Networks

Graph Sampling

GNN

Layer Sampling

Minibatch Training
Abstract: Graph Neural Networks (GNNs) have received significant attention recently, but training them at a large scale remains a challenge.
Mini-batch training coupled with sampling is used to alleviate this challenge.
However, existing approaches either suffer from the neighborhood explosion phenomenon or have suboptimal performance. 
To address these issues, we propose a new sampling algorithm called LAyer-neighBOR sampling (LABOR). 
It is designed to be a direct replacement for Neighbor Sampling (NS) with the same fanout hyperparameter while sampling up to 7 times fewer vertices, without sacrificing quality.
By design, the variance of the estimator of each vertex matches NS from the point of view of a single vertex.
Moreover, under the same vertex sampling budget constraints, LABOR converges 
faster than existing layer sampling approaches and can use up to 112 times larger batch sizes compared to NS.

Layer-Neighbor Sampling - Defusing Neighborhood Explosion in GNNs

Abstract

Graph Neural Networks (GNNs) have received significant attention recently, but training them at a large scale remains a challenge. Mini-batch training coupled with sampling is used to alleviate this challenge. However, existing approaches either suffer from the neighborhood explosion phenomenon or have suboptimal performance. To address these issues, we propose a new sampling algorithm called LAyer-neighBOR sampling (LABOR). It is designed to be a direct replacement for Neighbor Sampling (NS) with the same fanout hyperparameter while sampling up to 7 times fewer vertices, without sacrificing quality. By design, the variance of the estimator of each vertex matches NS from the point of view of a single vertex. Moreover, under the same vertex sampling budget constraints, LABOR converges faster than existing layer sampling approaches and can use up to 112 times larger batch sizes compared to NS.

1 Introduction

Graph Neural Networks (GNN) [Hamilton et al., 2017, Kipf and Welling, 2017] have become de facto models for representation learning on graph-structured data. Hence they have started being deployed in production systems [Ying et al., 2018, Niu et al., 2020]. These models iteratively update the node embeddings by passing messages along the direction of the edges in the given graph with nonlinearities between different layers. With l layers, the computed node embeddings contain information from the l -hop neighborhood of the seed vertex.

In the production setting, the GNN models need to be trained on billion-scale graphs [Ching et al., 2015, Ying et al., 2018]. The training of these models takes hours to days even on distributed systems [Zheng et al., 2022b,a]. As in general Deep Neural Networks (DNN), it is more efficient to use mini-batch training [Bertsekas, 1994] on GNNs, even though it is a bit trickier in this case. The node embeddings in GNNs depend recursively on their set of neighbors' embeddings, so when there are l layers, this dependency spans the l -hop neighborhood of the node. Real-world graphs usually have a very small diameter and if l is large, the l -hop neighborhood may very well span the entire graph, also known as the Neighborhood Explosion Phenomenon (NEP) [Zeng et al., 2020].

To solve these issues, researchers proposed sampling a subgraph of the l -hop neighborhood of the nodes in the batch. There are mainly three different approaches: Node-based, Layer-based and Subgraph-based methods. Node-based sampling methods [Hamilton et al., 2017, Chen et al., 2018a, Liu et al., 2020, Zhang et al., 2021] sample independently and recursively for each node. It was noticed that node-based methods sample subgraphs that are too shallow, i.e., with a low ratio of number of edges to nodes. Thus layer-based sampling methods were proposed [Chen et al., 2018b,

Zou et al., 2019, Huang et al., 2018, Dong et al., 2021], where the sampling for the whole layer is done collectively. On the other hand subgraph sampling methods [Chiang et al., 2019, Zeng et al., 2020, Hu et al., 2020b, Zeng et al., 2021, Fey et al., 2021, Shi et al., 2023] do not use the recursive layer by layer sampling scheme used in the node- and layer-based sampling methods and instead tend to use the same subgraph for all of the layers. Some of these sampling methods take the magnitudes of embeddings into account [Liu et al., 2020, Zhang et al., 2021, Huang et al., 2018], while others, such as Chen et al. [2018a], Cong et al. [2021], Fey et al. [2021], Shi et al. [2023], cache the historical embeddings to reduce the variance of the computed approximate embeddings. There are methods of sampling from a vertex cache Dong et al. [2021] filled with popular vertices. Most of these approaches are orthogonal to each other and they can be incorporated into other sampling algorithms.

Node-based sampling methods suffer the most from the NEP but they guarantee a good approximation for each embedding by ensuring each vertex gets k neighbors which is the only hyperparameter of the sampling algorithm. Layer-based sampling methods do not suffer as much from the NEP because the number of vertices sampled is a hyperparameter but they can not guarantee that each vertex approximation is good enough and also their hyperparameters are hard to reason with, the number of nodes to sample at each layer depends highly on the graph structure (as the numbers in Table 2 show). Moreover, Table 4 in Chen et al. [2023] shows that they can be extremely sensitive to the chosen hyperparameters and final task performance can differ by up to 15% between choosing a fixed number or doubling the number of sampled vertices at each layer. It is also unclear why one should settle with doubling; tripling, or quadrupling could potentially improve the performance of the final trained model even further. Subgraph sampling methods usually sample sparser subgraphs compared to their node- and layer-based counterparts. Hence, in this paper, we focus on the nodeand layer-based sampling methods and combine their advantages. The major contributions of this work can be listed as follows:

We propose the use of Poisson Sampling for GNNs, taking advantage of its lower variance and computational efficiency against sampling without replacement. Applying it to the existing layer sampling method LADIES, we get the superior PLADIES method outperforming the former by up to 2% in terms of F1-score. Moreover, PLADIES is linear time, improving upon the recently proposed quadratic time debiasing technique in Chen et al. [2023].

Wepropose a new sampling algorithm called LABOR, combining the advantages of neighbor and layer sampling approaches using Poisson Sampling. LABOR correlates the sampling procedures of the given set of seed nodes so that the sampled vertices from different seeds have a lot of overlap, resulting in a 7 × and 4 × reduction in the number of vertices and edges sampled compared to NS, respectively. Furthermore, LABOR can sample up to 13 × fewer edges compared to LADIES.

We experimentally verify our findings and show that our proposed sampling algorithm LABOR outperforms both neighbor and layer sampling approaches. LABOR can enjoy a batch size of up to 112 × larger than NS while sampling the same number of vertices.

2 Background

Graph Neural Networks: Given a directed graph G = ( V, E ) , where V and E ⊂ V × V are vertex and edge sets respectively, ( t → s ) ∈ E denotes an edge from a source vertex t ∈ V to a destination vertex s ∈ V , and A ts denotes the corresponding edge weight if provided. If we have a batch of seed vertices S ⊂ V , let us define l -hop neighborhood N l ( S ) for the incoming edges as follows:

Let us also define the degree d s of vertex s as d s = | N ( s ) | . To simplify, let's assume uniform edge weights, A ts = 1 , ∀ ( t → s ) ∈ E . Then, our goal is to estimate the following for each vertex s ∈ S , where H ( l -1) t is defined as the embedding of the vertex t at layer l -1 , and W ( l -1) is the trainable weight matrix at layer l -1 , and σ is the nonlinear activation function [Hamilton et al., 2017]:

Exact Stochastic Gradient Descent: If we have a node prediction task and V t ⊆ V is the set of training vertices, y s , s ∈ V t are the labels of the prediction task, and ℓ is the loss function for

the prediction task, then our goal is to minimize the following loss function: 1 | V t | ∑ s ∈ V t ℓ ( y s , Z l s ) . Replacing V t in the loss function with S ⊂ V t for each iteration of gradient descent, we get stochastic gradient descent for GNNs. However with l layers, the computation dependency is on N l ( S ) , which reaches large portion of the real world graphs, i.e. | N l ( S ) | ≈ | V | , making each iteration costly both in terms of computation and memory.

Neighbor Sampling: Neighbor sampling approach was proposed by Hamilton et al. [2017] to approximate Z ( l ) s for each s ∈ S with a subset of N l ( S ) . Given a fanout hyperparameter k , this subset is computed recursively by randomly picking k neighbors for each s ∈ S from N ( s ) to form the next layer S 1 , that is a subset of N 1 ( S ) . If d s ≤ k , then the exact neighborhood N ( s ) is used. For the next layer, S 1 is treated as the new set of seed vertices and this procedure is applied recursively.

Revisiting LADIES, Dependent Layer-based Sampling: From now on, we will drop the layer notation and focus on a single layer, and also ignore the nonlinearities. Let us define M t = H t W as a shorthand notation. Then our goal is to approximate:

If we assign probabilities π t > 0 , ∀ t ∈ N ( S ) and normalize it so that ∑ t ∈ N ( S ) π t = 1 , then use sampling with replacement to sample T ⊂ N ( S ) with | T | = n , where n is the number of vertices to sample given as input to the LADIES algorithm and T is a multi-set possibly with multiple copies of the same vertices, and let ˜ d s = | T ∩ N ( s ) | which is the number of sampled vertices for a given vertex s , we get the following two possible estimators for each vertex s ∈ S :

Note that H ′ s in (4a) is the Thompson-Horvitz estimator and the H ′′ s in (4b) is the Hajek estimator. For a comparison between the two and how to get an even better estimator by combining them, see Khan and Ugander [2021]. The formulation in the LADIES paper uses H ′ s , but it proposes to row-normalize the sampled adjacency matrix, meaning they use H ′′ s in their implementation. However, analyzing the variance of the Thompson-Horvitz estimator is simpler and its variance serves as an upper bound for the variance of the Hajek estimator when | M t | and π t are uncorrelated [Khan and Ugander, 2021, Dorfman, 1997], which we assume to be true in our case. Note that the variance analysis is simplified to be element-wise for all vectors involved.

Since we do not have access to the computed embeddings and to simplify the analysis, we assume that Var( M t ) = 1 from now on. One can see that Var( H ′ s ) is minimized when π t = p, ∀ t → s under the constraint ∑ t → s π t ≤ pd s for some constant p ∈ [0 , 1] , hence any deviation from uniformity increases the variance. The variance is also smaller the larger ˜ d s is. However, in theory and in practice, there is no guarantee that each vertex s ∈ S will get any neighbors in T , not to mention equal numbers of neighbors. Some vertices will have pretty good estimators with thousands of samples and very low variances, while others might not even get a single neighbor sampled. For this reason, we designed LABOR so that every vertex in S will sample enough neighbors in expectation.

While LADIES is optimal from an approximate matrix multiplication perspective [Chen et al., 2023], it is far from optimal in the case of nonlinearities and multiple layers. Even if there is a single layer, then the used loss functions are nonlinear. Moreover, the existence of nonlinearities in-between layers and the fact that there are multiple layers exacerbates this issue and necessitates that each vertex gets a good enough estimator with low enough variance. Also, LADIES gives a formulation using sampling with replacement instead of without replacement and that is sub-optimal from the variance perspective while its implementation uses sampling without replacement without taking care of the bias created thereby. In the next section, we will show how all of these problems are addressed by our newly proposed Poisson sampling framework and LABOR sampling.

3 Proposed Layer Sampling Methods

Node-based sampling methods suffer from sampling too shallow subgraphs leading to NEP in just a few hops (e.g., see Table 2). Layer sampling methods Zou et al. [2019] attempt to fix this by sampling a fixed number of vertices in each layer, however they can not ensure that the estimators for the vertices are of high quality, and it is hard to reason how to choose the number of vertices to sample in each layer. LADIES Zou et al. [2019] proposes using the same number for each layer while papers evaluating it found it is better to sample an increasing number of vertices in each layer Liu et al. [2020], Chen et al. [2023]. There is no systematic way to choose how many vertices to sample in each layer for the LADIES method, and since each graph has different density and connectivity structure, this choice highly depends on the graph in question. Therefore, due to its simplicity and high quality results, Neighbor Sampling currently seems to be the most popular sampling approach and there exists high quality implementations on both CPUs and GPUs in the popular GNN frameworks Wang et al. [2019], Fey and Lenssen [2019].

We propose a new approach that combines the advantages of layer and neighbor sampling approaches using a vertex-centric variance-based framework, reducing the number of sampled vertices drastically while ensuring the training quality does not suffer and matches the quality of neighbor sampling. Another advantage of our method is that the user only needs to choose the batch size and the fanout hyperparameters as in the Neighbor Sampling approach, the algorithm itself then samples the minimum number of vertices in the later layers in an unbiased way while ensuring each vertex gets enough neighbors and a good approximation.

We achieve all the previously mentioned good properties with the help of Poisson Sampling. So, the next section will demonstrate applying Poisson Sampling to Layer Sampling, and then we will show how the advantages of Layer and Neighbor Sampling methods can be combined into LABOR while getting rid of their cons altogether.

3.1 Poisson Layer Sampling (PLADIES)

In layer sampling, the main idea can be summarized as individual vertices making correlated decisions while sampling their neighbors, because in the end if a vertex t is sampled, all edges into the seed vertices S , i.e., t → s , s ∈ S , are added to the sampled subgraph. This can be interpreted as vertices in S making a collective decision on whether to sample t , or not.

The other thing to keep in mind is that the existing layer sampling methods use sampling with replacement when doing importance sampling with unequal probabilities, because it is nontrivial to compute the inclusion probabilities in the without replacement case. The Hajek estimator in the without replacement case with equal probabilities becomes:

and it has the variance:

Let us show how one can do layer sampling using Poisson sampling (PLADIES). Given n ∈ Z + and unnormalized probabilities π t ∈ [0 , ∞ ) , ∀ t ∈ N ( S ) , we can ensure that ∑ t ∈ N ( S ) π t = n by finding a solution of ∑ t ∈ N ( S ) min(1 , cπ t ) = n for c ∈ R + in linear time and redefining π t ← min(1 , cπ t ) so that π t ∈ [0 , 1] . Then we include t ∈ N ( S ) in our sample T with probability π t by flipping a coin for it, i.e., we sample r t ∼ U (0 , 1) and include t ∈ T if r t ≤ π t . In the end, E [ | T | ] = n and we can still use the Hajek estimator H ′′ s or the Horvitz Thomson estimator H ′ s to estimate H s . Doing layer sampling this way is unbiased by construction and achieves the same goal in linear time in contrast to the quadratic time debiasing approach explained in Chen et al. [2023], further discussion can be found in Appendix A.7. The variance then approximately becomes [Williams et al., 1998], see Appendix A.1 for a derivation:

One can notice that the minus term 1 d s enables the variance to converge to 0 , if all π t = 1 and we get the exact result. However, in the sampling with replacement case, the variance goes to 0 only as the sample size goes to infinity.

3.2 LABOR: Layer Neighbor Sampling

The design philosophy of LABOR Sampling is to create a direct alternative to Neighbor Sampling while incorporating the advantages of layer sampling. Mimicking Layer Sampling with Poisson Sampling in Section 3.1 still has the disadvantage that ˜ d s varies wildly for different s . To overcome this and mimic Neighbor Sampling where E [ ˜ d s ] = min( d s , k ) , where k is a given fanout hyperparameter, we proceed as follows: for given π t ≥ 0 , ∀ t ∈ N ( S ) denoting unnormalized probabilities, for a given s , let us define c s as the quantity satisfying the following equality if k < d s , otherwise c s = max t → s 1 π t :

Note that 1 k -1 d s is the variance when π t = k d s , ∀ t ∈ N ( s ) so that E [ ˜ d s ] = k . Also note that:

meaning that the variance target we set through (9) is equal to Neighbor Sampling's variance in (7) after calibrating with d s d s -1 Ohlsson [1998] and it will result in E [ ˜ d s ] ≥ k with strict equality in the uniform probability case. Then each vertex s ∈ S samples t → s with probability c s π t . To keep the collective decision making, we sample r t ∼ U (0 , 1) , ∀ t ∈ N ( S ) and vertex s samples vertex t if and only if r t ≤ c s π t . Note that if we use a uniform random variable for each edge r ts instead of each vertex r t , and if π is uniformly initialized, then we get the same behavior as Neighbor Sampling.

3.2.1 Importance Sampling

Given the sampling procedure above, one wonders how different choices of π ≥ 0 will affect | T | , the total number of unique vertices sampled. We can compute its expected value as follows:

In particular, we need to minimize E [ | T | ] :

Note that for any given π ≥ 0 , E [ | T | ] is the same for any vector multiple xπ, x ∈ R + , meaning the objective function is homogeneous of degree 0 .

3.2.2 Computing c and π ∗

c s was defined to be the scalar satisfying the following equality involving the variance of the estimator of H s :

If we rearrange the terms, we get:

One can see that the left-hand side of the equality is monotonically decreasing with respect to c s ≥ 0 . Thus one can use binary search to find the c s satisfying the above equality to any precision needed. But we opt to use the following iterative algorithm to compute it:

This iterative algorithm converges in at most d s steps and the convergence is exact and monotonic from below. One can also implement it in linear time O ( d s ) if { π t | t → s } is sorted and making use of precomputed prefix sum arrays. Note that c = c ( π ) , meaning that c is a function of the given probability vector π . To compute π ∗ , we use a similar fixed point iteration as follows:

Thus, we alternate between computing c = c ( π ) , meaning c is computed with the current π , and updating π with the computed c values, see the lines 5-10 in Algorithm 1. Each step of this iteration is guaranteed to lower the objective function value in (12) until convergence to a fixed point, see Appendix A.3. Modified formulation for a given weight matrix A ts is discussed in Appendix A.5. Finally, Algorithm 1 summarizes LABOR including the actual sampling (lines 17-19) and computation of the edge weights (lines 20-22) due to the use of importance sampling.

3.3 Choosing how many neighbors to sample

The variance of Poisson Sampling when π t = k d s is 1 k -1 d s . One might question why we are trying to match the variance of Neighbor Sampling and choose to use a fixed fanout for all the seed vertices. In the uniform probability case, if we have already sampled some set of edges for the vertices in S , and want to sample one more edge, then we should sample the new edge for the vertex s , whose variance would decrease the most. If vertex s currently has ˜ d s sampled edges, then sampling one more edge for it would improve its variance from 1 ˜ d s -1 d s to 1 1+ ˜ d s -1 d s . Since the derivative of the variance with respect to ˜ d s is monotonic, we can reason about the marginal improvements by comparing their derivatives, which is:

Notice that the derivative does not depend on the degree d s of the vertex s at all, and the greater the magnitude of the derivative, the more the variance of a vertex improves by sampling one more edge. Thus, choosing any vertex s with least number of edges sampled would work for us, that is s = arg min s ′ ∈ S ˜ d s ′ . In light of this observation, one can see that it is optimal to sample an equal number of edges for each vertex in S . This is one of the reasons LADIES is not efficient with respect to the number of edges it samples. On graphs with skewed degree distributions, it samples thousands of edges for some seed vertices, which contribute very small amounts to the variance of the estimator since it is already very close to 0 .

3.4 Sampling a fixed number of neighbors

To make LABOR-0 even more similar to Neighbor Sampling, one can easily resort to Sequential Poisson Sampling by Ohlsson [1998] if one wants ˜ d s = min( k, d s ) instead of E [ ˜ d s ] = min( k, d s ) . Given π t , c s and r t , we pick the ˜ d s = min( k, d s ) smallest vertices t → s with respect to r t c s π t , which can be computed in expected linear time by using the quickselect algorithm [Hoare, 1961]. 4

4 Experiments

In this section, we empirically evaluate the performance of each method in the node-prediction setting on the following datasets: reddit [Hamilton et al., 2017], products [Hu et al., 2020a], yelp, flickr [Zeng et al., 2020]. Details about these datasets are given in Table 1. We compare our proposed LABOR variants LABOR-0, LABOR-1 and LABOR-*, where 0 , 1 , ∗ stand for the number of fixed point iterations applied to optimize (12), respectively, together with PLADIES (see Section 3.1), against the baseline sampling methods: Neighbor Sampling (NS) and LADIES. We do not include Fast-GCN in our comparisons as it is super-seeded by the LADIES paper. The works of Liu et al. [2020], Zhang et al. [2021], Huang et al. [2018], Cong et al. [2021], Dong et al. [2021] are not included in the comparisons because they either take into account additional information such as historical embeddings or their magnitudes or they have an additional sampling structure such as a vertex cache to sample from. Also the techniques in these papers are mostly orthogonal to the sampling problem and algorithms discussed in this paper, hence our proposed LABOR and PLADIES methods can be used together with them. However, we leave this investigation to future work.

We evaluate all the methods on the GCN model in (2) with 3 layers, with 256 hidden dimension and residual skip connections enabled. We use the Adam optimizer [Kingma and Ba, 2014] with 0 . 001 learning rate. We implemented LABOR variants and PLADIES in DGL 5 [Wang et al., 2019], and carried out our experiments using DGL with the Pytorch backend [Paszke et al., 2019], URLs are provided in Appendix A.8. Experiments were repeated 100 times and averages are presented.

4.1 Baseline Comparison of Vertex/Edge Efficiency

LABOR comparison against NS: In this experiment, we set the batch size to 1,000 and the fanout k = 10 for LABOR and NS methods to see the difference in the sizes of the sampled subgraphs and also how vertex/edge efficient they are. In Figure 1, we can see that LABOR variants outperform NS in both vertex and edge efficiency metrics. Table 2 shows the difference of the sampled subgraph sizes in each layer. One can see that on reddit, LABOR-* samples 6 . 9 × fewer vertices in the 3rd layer leading much faster convergence in terms of cumulative number of sampled vertices. On the flickr dataset however, LABOR-* samples only 1 . 3 × fewer vertices. The amount of difference depends on two factors. The first is the amount of overlap of neighbors among the vertices in S . If the neighbors of vertices in S did not overlap at all, then one obviously can not do better than NS. The second is the average degree of the graph. With a fanout of 10 , both NS and LABOR has to copy the whole neighborhood of a vertex s with degree d s ≤ 10 . Thus for such graphs, it is expected that the difference will be insignificant. The average degree of the flickr is 10 . 09 (Table 1), and thus there is only a small difference between LABOR and NS.

As seen in Figure 1 and Table 2, LABOR-0 reduces both the number of vertices and edges sampled. On the other hand, when importance sampling is enabled, the number of vertices sampled goes down while number of edges sampled goes up. This is because when importance sampling is used, inclusion probabilities become nonuniform and it takes more edges per seed vertex to get a good approximation (see Equation (9)). Thus, LABOR-0 leads the pack when it comes to edge efficiency.

Layer sampling algorithm comparison: The hyperparameters of LADIES and PLADIES were picked to match LABOR-* so that all methods have the same vertex sampling budget in each layer (see Table 2). Figure 1 shows that, LADIES and PLADIES perform almost identically in terms of convergence, on all but the flickr dataset where there is a big difference between the two in favor of PLADIES. We also see that LABOR variants outperform LADIES variants on products, yelp, and flickr in terms of vertex efficiency with the exception of reddit, where the best-performing algorithm is PLADIES proposed by us. Regarding edge efficiency, however, we see that LADIES variants are inefficient and all LABOR variants outperform LADIES variants across all the datasets by up to 13 × .

Comparing LABOR variants: Looking at Table 2, we can see that LABOR-0 has the best runtime performance across all datasets. This is due to 1) sampling fewer edges, 2) not having the fixed point iteration overhead, compared to the other LABOR variants. By design, all LABOR variants should

have the same convergence curves, as seen in Appendix A.2 in Figure 3. Then, the decision of which variant to use depends on one factor: feature access speed. If vertex features were stored on a slow storage medium (such as, on host memory accessed over PCI-E), then minimizing number of sampled vertices would become the highest priority, in which case, one should pick LABOR-*. Depending on the relative vertex feature access performance and the performance of the training processor, one can choose to use LABORj , the faster feature access, the lower the j .

4.2 LABOR vs NS under vertex sampling budgets

In this experiment, we set a limit on the number of sampled vertices and modify the batch size to match the given vertex budget. The budgets used were picked around the same magnitude with numbers in the Table 2 in the | V 3 | column and can be found in Table 1. Figure 2 displays the result of this experiment. Table 3 shows that the more vertex efficient the sampling method is, the larger batch size it can use during training. Number of sampled vertices is not a function of the batch size for the LADIES algorithm so we do not include it in this comparison. All of the experiments were repeated 100 times and their averages were plotted, that is why our convergence plots are smooth and differences are clear. The most striking result in this experiment is that there can be up to 112 × difference in batch sizes of LABOR-* and NS algorithms on the reddit dataset, which translates into faster convergence as the training loss and validation F1-score curves in Figure 2 show.

4.3 Importance Sampling, Fixed Point Iterations

In this section, we look at the convergence behavior of the fixed point iterations described in Section 3.2.2. Table 4 shows the number of sampled vertices in the last layer with respect to the number of fixed point iterations applied. In this table, the * stands for applying the fixed point iterations until

convergence, and convergence occurs in at most 15 iterations in practice before the relative change in the objective function is less than 10 -4 . One can see that most of the reduction in the objective function (12) occurs after the first iteration, and the remaining iterations have diminishing returns. Full convergence can save from 14% -33% depending on the dataset. The monotonically decreasing numbers provide empirical evidence for the presented proof in Appendix A.3.

5 Conclusions

In this paper, we introduced LABOR sampling, a novel way to combine layer and neighbor sampling approaches using a vertex-variance centric framework. We then transform the sampling problem into an optimization problem where the constraint is to match neighbor sampling variance for each vertex while sampling the fewest number of vertices. We show how to minimize this new objective function via fixed-point iterations. On datasets with dense graphs like Reddit, we show that our approach can sample a subgraph with 7 × fewer vertices without degrading the batch quality. We also show that compared to LADIES, LABOR converges faster with same sampling budget.

References

D.P. Bertsekas. Incremental least squares methods and the extended kalman filter. In Proceedings of 1994 33rd IEEE Conference on Decision and Control , volume 2, pages 1211-1214 vol.2, 1994. doi: 10.1109/CDC.1994.411166.

Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In International Conference on Learning Representations , 2022. URL https://openreview.net/ forum?id=F72ximsx7C1 .

Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. 35th International Conference on Machine Learning, ICML 2018 , 3:1503-1532, 2018a.

Jie Chen, Tengfei Ma, and Cao Xiao. FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling. jan 2018b. URL http://arxiv.org/abs/1801.10247 .

Yifan Chen, Tianning Xu, Dilek Hakkani-Tur, Di Jin, Yun Yang, and Ruoqing Zhu. Calibrate and debias layer-wise sampling for graph convolutional networks. Transactions on Machine Learning Research , 2023. URL https://openreview.net/forum?id=JyKNuoZGux .

Wei Lin Chiang, Yang Li, Xuanqing Liu, Samy Bengio, Si Si, and Cho Jui Hsieh. Cluster-GCN: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 257-266. Association for Computing Machinery, jul 2019. doi: 10.1145/3292500.3330925.

Avery Ching, Sergey Edunov, Maja Kabiljo, Dionysios Logothetis, and Sambavi Muthukrishnan. One trillion edges: Graph processing at facebook-scale. Proc. VLDB Endow. , 8(12):1804-1815, aug 2015. doi: 10.14778/2824032.2824077. URL https://doi.org/10.14778/2824032. 2824077 .

Weilin Cong, Morteza Ramezani, and Mehrdad Mahdavi. On the Importance of Sampling in Training GCNs: Tighter Analysis and Variance Reduction. 2021. URL http://arxiv.org/abs/2103. 02696 .

Alexander I Cowen-Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Hao Jianye, Jun Wang, and Haitham Bou Ammar. An empirical study of assumptions in bayesian optimisation. arXiv preprint arXiv:2012.03826 , 2020.

Jialin Dong, Da Zheng, Lin F. Yang, and George Karypis. Global Neighbor Sampling for Mixed CPU-GPU Training on Giant Graphs. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 289-299, 2021. doi: 10.1145/3447548.3467437.

Alan H Dorfman. The Hajek Estimator Revisited. (4):760-765, 1997. URL http://www.asasrms. org/Proceedings/papers/1997_130.pdf .

Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric, 2019. URL https://arxiv.org/abs/1903.02428 .

Matthias Fey, Jan E. Lenssen, Frank Weichert, and Jure Leskovec. Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 3294-3304. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/fey21a.html .

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 5dd9db5e033da9c6fb5ba83c7a7ebea9-Paper.pdf .

A. R. Hoare. Algorithm 65: Find. Commun. ACM , 4(7):321-322, jul 1961. doi: 10.1145/366622. 366647. URL https://doi.org/10.1145/366622.366647 .

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in Neural Information Processing Systems , 2020-Decem(NeurIPS):1-34, 2020a.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous Graph Transformer. The Web Conference 2020 - Proceedings of the World Wide Web Conference, WWW 2020 , pages 2704-2710, 2020b. doi: 10.1145/3366423.3380027.

Wenbing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. Adaptive sampling towards fast graph representation learning. Advances in Neural Information Processing Systems , 2018-Decem(Nips): 4558-4567, 2018.

Samir Khan and Johan Ugander. Adaptive normalization for IPW estimation. pages 1-31, 2021. URL http://arxiv.org/abs/2106.07695 .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL https://arxiv.org/abs/1412.6980 .

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings , pages 1-14, 2017.

Ziqi Liu, Zhengwei Wu, Zhiqiang Zhang, Jun Zhou, Shuang Yang, Le Song, and Yuan Qi. Bandit samplers for training graph neural networks. Advances in Neural Information Processing Systems , 2020-Decem, 2020.

Xichuan Niu, Bofang Li, Chenliang Li, Rong Xiao, Haochuan Sun, Hongbo Deng, and Zhenzhong Chen. A dual heterogeneous graph attention network to improve long-tail performance for shop search in e-commerce. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '20, page 3405-3415, 2020. doi: 10.1145/3394486. 3403393.

E Ohlsson. Sequential poisson sampling. Journal of Official Statistics-Stockholm, 14(2):149-162, 1998.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , 2019.

Zhihao Shi, Xize Liang, and Jie Wang. LMC: Fast training of GNNs via subgraph sampling with provable convergence. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=5VBBA91N6n .

Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks, 2019. URL https://arxiv.org/abs/1909.01315 .

Michael S Williams, Hans T Schreuder, and Gerardo H Terrazas. Poisson Sampling - The Adjusted and Unadjusted Estimator Revisited. page 12, 1998.

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 974-983, 2018. doi: 10.1145/3219819.3219890.

Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. GraphSAINT: Graph sampling based inductive learning method. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=BJe8pkHFwS .

Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, and Ren Chen. Decoupling the depth and scope of graph neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id= d0MtHWY0NZ .

Qingru Zhang, David Wipf, Quan Gan, and Le Song. A Biased Graph Neural Network Sampler with Near-Optimal Regret. (NeurIPS):1-25, 2021. URL http://arxiv.org/abs/2103.01089 .

Che Zheng, Hongzhi Chen, Yuxuan Cheng, Zhezheng Song, Yifan Wu, Changji, Li, James Cheng, Han Yang, and Shuai Zhang. Bytegnn: Efficient graph neural network training at large scale. Proc. VLDB Endow. , 15:1228-1242, 2022a.

Da Zheng, Xiang Song, Chengru Yang, Dominique LaSalle, and George Karypis. Distributed hybrid cpu and gpu training for graph neural networks on billion-scale heterogeneous graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4582-4591, 2022b.

Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. Layer-dependent importance sampling for training deep and large graph convolutional networks. Advances in Neural Information Processing Systems , 32(NeurIPS), 2019.

A Appendix

A.1 Derivation of Poisson Sampling variance

Following is a derivation of Equation (8) for the Horvitz-Thomson estimator. Note that we assume Var( M t ) = 1 , ∀ t . Also, Var( ✶ [ r t ≤ π t ]) is the variance of the bernoulli trial with probability π t and is equal to π t (1 -π t ) .

A.2 Baseline Comparison of Vertex/Edge Efficiency (cont.)

Here, we show another version of Figure 1, replacing the x-axis with the number of training iterations, see Figure 3. We would like to point out that despite NS and LABOR variants sampling drastically

different number of vertices/edges as shown in Table 2 (up to 7 × fewer vertices), their convergence behavior is more or less the same, even indistinguishable on Yelp and Flickr.

A.3 Monotonic convergence proof of the fixed point iterations

Given any π (0) > 0 and one iteration to get π (1) and one more iteration using c ( π (1) ) to get π (2) , then we have the following observation:

Now, note that for a given t ∈ N ( s ) , max t → s ′ c s ′ ( π (0) ) ≥ c s ( π (0) ) since s ∈ { s ′ | t → s ′ } . This implies that max t → s ′ c s ′ ( π (0) ) π (0) t ≥ c s ( π (0) ) π (0) t . Note that for π ′ t = c s ( π (0) ) π (0) t , ∀ t → s , we have c s ( π ′ ) = 1 for any given s . Since max t → s ′ c s ′ ( π (0) ) π (0) t ≥ c s ( π (0) ) π (0) t = π ′ t , ∀ t → s , this let's us conclude that the c s ( π (1) ) ≤ 1 , because the expression is monotonically increasing with respect to any of the π t . By induction, this means that c ( i ) s ≤ 1 , ∀ i ≥ 1 . Since π ( i ) t = π (0) t ∏ i -1 j =0 max t → s ′ c s ′ ( π ( i -1) ) , π ( i ) t is monotonically decreasing. This means that the objective value in (12) is also monotonically decreasing and is clearly bounded from below by 0 . Any monotonically decreasing sequence bounded from below has to converge, so our fixed point iteration procedure is convergent as well.

The intuition behind the proof above is that after updating π via (18), the probability of each edge t → s : π t c s goes up because the update takes the maximum c s over all possible s . This means each vertex gets higher quality estimators than the set variance target so we now have room to reduce the number of vertices sampled by choosing the appropriate c s ≤ 1 . To summarize, π update step increases the quality of the batch which in turn let's the c s step to reduce the total number of vertices sampled.

A.4 GATv2 runtime performance

We utilize the GATv2 model Brody et al. [2022] and show the runtime performance of different samplers in Table 5. All the hyperparameters were kept same as Section 4.1 and the number of attention heads was set to 8. The runtimes correlate with the | E 3 | column in Table 2 for the GATv2 model as the computation and memory requirements depend on the number of edges. That is why the LADIES variants went out-of-memory on the reddit and products datasets.

A.5 Extension to the weighted case

If the given adjacency matrix A has nonuniform weights, then we want the estimate the following:

where A ∗ s = ∑ t → s A ts . If we have a probability over each edge π ts , ∀ ( t → s ) ∈ E , then the variance becomes:

In this case, we can still aim to reach the same variance target v s = 1 k -1 d s or any given custom target v s ∈ R + by finding c s that satisfies the following equality:

In this case, the objective function becomes:

Optimizing the objective function above will result into minimizing the number of vertices sampled. Given any π ts > 0 , ∀ ( t → s ) , then the fixed point iterations proposed for the non-weighted case in (18) can be modified as follows:

A more principled way to choose v s in the weighted case is by following the argument presented in Section 3.3. There, the discussion revolves around the derivative of the variance with respect to the expected number of vertices sampled for a given seed vertex s . If we apply the same argument, then we get:

Then to compute the derivative of the variance with respect to the expected number of sampled vertices for a given seed vertex s , which is ∂v s ∂E [ ˜ d s ] , we can use the chain rule and get:

✶ Then, what one would do is to choose a constant C ( k ) as a function of the fanout parameter k and set ∂v s ∂E [ ˜ d s ] = C ( k ) and solve for c s . C ( k ) would be a negative quantity whose absolute value decreases as k increases. It would probably look like C ( k ) = -C ′ k 2 for some constant C ′ > 0 . We leave this as future work.

A.6 Convergence speed with respect to wall time

In this section, we perform hyperparameter optimization on Neighbor Sampler and Labor Sampler so that the training converges to a target validation accuracy as fast as possible. We leave LADIES out of this experiment because it is too slow as can be seen in the last column of Table 2. We ran this experiment on an A100 GPU and stored the input features on the main memory, which were accessed over the PCI-e directly during training by pinning their memory. This kind of training scenario is commonly used when training on large datasets whose input features don't fit in the GPU memory. We use larger of the two datasets we have for this experiment, products and yelp. For products, the validation accuracy target we set is 91.5%, and for yelp it is 60%. We tune the learning rate between [10 -4 , 10 -1 ] , the batch size between [2 10 , 2 15 ] and the fanout for each layer between [5 , 25] . We use the same model used in Section 4. For LABOR, we additionally tune over the number of importance sampling iterations i between [0 , 3] so that it can switch between LABORi and also a layer dependency boolean parameter that makes LABOR use the same random variates r t for different layers when enabled, which has the effect of increasing the overlap of sampled vertices across layers. We use the state-of-the-art hyperparameter tuner HEBO [Cowen-Rivers et al., 2020], the winning submission to the NeurIPS 2020 Black-Box Optimization Challenge, to tune the parameters of the sampling algorithm with respect to runtime required to reach the target validation accuracy with a timeout of 300 seconds, terminating the run if the configuration doesn't reach the target accuracy.

We let HEBO run overnight and collect the minimum runtimes required to achieve the target accuracies. For products, the fastest hyperparameter corresponding to the 38.2s runtime had fanouts (18 , 5 , 25) , batch size 10500 , learning rate 0 . 0145 , used LABOR-1, layer dependency False.

For Neighbor Sampler, the fastest hyperparameter corresponding to 43.82s runtime had fanouts (15 , 5 , 21) , batch size 12000 , learning rate 0 . 0144 .

For the Yelp dataset, the fastest hyperparameter corresponding to the 41.60s runtime had fanouts (6 , 5 , 7) , batch size 5400 , learning rate 0 . 000748 , used LABOR-1 and layer dependency True.

For Neighbor Sampler, the fastest hyperparameter corresponding to the 47.40s runtime had fanouts (5 , 6 , 6) , batch size 4600 , learning rate 0 . 000931 .

These results clearly indicate that training with LABOR is faster compared to Neighbor Sampling when it comes to time to convergence.

We run the same experiment with GraphSAINT [Zeng et al., 2020] using the DGL example code, both on ogbn-products and yelp using the same model architecture. We used their edge sampler, the edge sampling budget between [2 10 , 2 15 ] and learning rate between [10 -4 , 10 -1 ] . We disabled batch normalization to make it a fair comparison with NS and LABOR since the models they are using does not have batch normalization. We use HEBO to tune these hyperparameters to reach the set validation accuracy, and let it run overnight. The results show that HEBO was not able to find any configuration of the hyperparameters reaching 91.5% accuracy on products faster than 1500 seconds.

For the Yelp dataset, the fastest runtime to reach 60% accuracy was 92.2s, with an edge sampling budget of 12500 and learning rate 0 . 0214 .

A.7 Reducing debiased LADIES complexity from quadratic to linear time

Our proposed PLADIES algorithm in Section 3.1 is linear time and unbiased by construction via the use of Poisson Sampling, which has the same comparative advantages as sampling without replacement over sampling with replacement. However, LADIES authors Zou et al. [2019] formulate their sampling framework via sampling with replacement while their implementation uses sampling without replacement, without addressing the bias created thereby. Chen et al. [2023] concurrently noticed the same bias issue as us, and proposed a way to "de-bias" sampling without replacement by computing actual sampling probabilities in quadratic time (Chen et al. [2023], Section 5.5).

When one uses weighted sampling without replacement as in Chen et al. [2023], the sampling probability is actually not the weights provided initially and Chen et al. [2023] proposes a way to compute the actual probabilities and a way to de-bias them in quadratic time. There is no convergence difference regarding PLADIES and the de-bias procedure proposed in Chen et al. [2023], only the computational complexity of sampling is different. They also propose a way to flatten the probabilities and these flattened probabilities can be fed to PLADIES as well. So the flattening approach Chen et al. [2023] proposes is orthogonal to our work.

To give an example of why weighted sampling without replacement is biased, consider the probabilities [0.7, 0.2, 0.1]. Now, if we want to sample 2 items without replacement, then notice that the probability of the first item, 0.7, is more than 2 times the rest of the items, 0.3. Thus, the first item needs to be sampled always. But even if we always sample the first item, the second item is going to be either 0.2 or 0.1, the probability of which one it is going to be is [2/3, 1/3]. Thus, in the end, the probability of sampling individual items is [1, 2/3, 1/3]. However, these resulting probabilities are not linearly proportional to the probabilities we started with. Chen et al. [2023] fixes this in quadratic time. We propose to use Poisson Sampling instead. With Poisson sampling, we can have a constant c to adjust how many items we want to sample. We want 2 items in expectation, so 0 . 7 c +0 . 2 c +0 . 1 c = 2 . But the solution is c = 2 , and the probability of sampling the first item becomes 1.4, which is not possible. So, we need to solve for c in the following equation: min(0 . 7 c, 1) + min(0 . 2 c, 1) + min(0 . 1 c, 1) = 2 . The solution is c = 10 / 3 , and this can be found in linear time. The resulting probabilities become: min(0 . 7 c, 1) = 1 , min(0 . 2 c, 1) = 2 / 3 , min(0 . 1 c, 1) = 1 / 3 . Because we use these adjusted probabilities during GNN aggregation, our method is unbiased, and faster than Chen et al. [2023] due to the drastic computational complexity difference (quadratic vs linear).

A.8 Links to the experimental code

Experiments in Section 4 can be reproduced using https://github.com/dmlc/dgl/tree/ f971e25a4dff33d5d219dccf523e32c62360ffd2/examples/pytorch/labor . Our initial contribution to DGL can be found in https://github.com/dmlc/dgl/pull/4668 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the LABOR sampling algorithm to address the NEP problem in GNN. LABOR combines layer sampling (LADIES), neighbor sampling (NS), and Poisson sampling. In contrast to prior works, LABOR calculates the sampling probabilities for importance sampling “iteratively” for each layer. Moreover, the Poisson sampling method (PLADIES) enhances the existing LADIES method by using “sampling without replacement” instead of “sampling with replacement.” Also, the authors derive the hidden layer variance of PLADIES and find that, under certain conditions, the variance can converge to 0, showing better convergence properties. Experiments show that LABOR has better sampling efficiency, using only 1/7 vertex to achieve the same F1-score as NS and using 112 times larger batch sizes during training.\n\nSTRENGTHS:\n•\tLABOR has a better estimator of sampling with lower variances than prior work LADIES, which might not have a vertex with its neighbor among all sampled vertices.\n•\tLABOR has better vertex/edge efficiency than NS. As a result, LABOR can reach the same F1-score by using fewer sampled vertices under the same hyperparameter settings.\n\nWEAKNESSES:\nThe experimental evaluation is not sufficiently detailed.\n\nSome technical details are not clear.\n\nQUESTIONS:\nFigure 1 shows that the vertices efficiency of LADIES/PLADIES might be higher than LABOR in Reddit dataset. Why?\n\nLIMITATIONS:\nWhile the paper addresses an important topic, it has several issues that need to be addressed before publication:\n\n•\tLABOR outperforms NS in a graph with a high average degree (same as the assumption in the NEP problem), such as the Reddit dataset (average degree 493) shown in Table 2. However, when the average degree of the dataset is low (e.g., flickr: 10.09), the performance gap between LABOR and NS will be smaller.\n•\tLABOR is more often compared with NS in the paper, but LABOR is highly based on LADIES/layer sampling method. For example, the vertices efficiency only compared to NS in section 4.2, but Figure 1 shows that the vertices efficiency of LADIES/PLADIES might be higher than LABOR in Reddit dataset.\n•\tNeighbor Sampling (NS) is an important comparison subject in this paper, but it is not detailed in both Introduction and Background. The background simply states, “Neighbor Sampling: The neighbor sampling approach was proposed by Hamilton et al. [2017]” with a citation to GraphSage. Yet, section 3 describes that “Neighbor Sampling currently seems to be the most popular sampling approach.” The latter statement does not seem to refer to GraphSage.\n•\tThe paper needs to report the improvement in the total training time explicitly, as the abstract only mentions “7 times fewer vertices” and “use up to 112 times larger batch sizes compared to NS”.\n•\tTable 2 does not give detailed hyperparameter settings. For example, how were the hyperparameters obtained? what’s the range for hyperparameter sweeping? LADIES paper shows a 5 times total time difference by only adjusting hyperparameters – LADIES (64) and LADIES (512) on the Cora dataset. Moreover, details need to be provided in line241-242 and Figure2’s caption – “The hyperparameters of LADIES and PLADIES were picked to roughly match LABOR-”\n•\t“same sampling budget” needs to be defined more clearly in Table 2.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper proposes a new sampling algorithm called LAyer-neighBOR sampling (LABOR). The authors show the variance of the estimator is controlled. Experiments show this method is much more efficient in sampling and faster in convergence.

STRENGTHS:
1. The paper is well-written and clear.
2. Some analysis is provided. The experiments on some large graph datasets verify the advantages of the proposed method.

WEAKNESSES:
1. Only variance reduction analysis is provided. Theoretical convergence analysis or generalization analysis is lacking.
2. This paper mainly compares with LADIES, which is a work in 2019. I don't think it is still a state-of-the-art work about graph sampling that is appropriate for comparison. Some more recent works should be covered in experiments for comparison.

QUESTIONS:
1. The proposed method allows a larger batch size. This can make convergence faster. However, a larger batch size may not lead to a better generalization. This makes the statement of a faster convergence weaker.

LIMITATIONS:
There is no potential negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper improves the previous layer-wise importance sampling algorithm in GNN training by: 1) introduce poison sampling for debiasing; 2) it modifies the sampling importance by incorporating the probability of node-wise sampling, to make sure all nodes have similar neighbor nodes.\n\nIt shows that the variance of each sampled nodes is similar to that of node-wise sampling, while enjoying some benefits of efficiency of layer-wise sampling. It shows that with similar budget, the proposed method converges faster and more memory efficient.\n\nSTRENGTHS:\n1. It systemetically studeis the limitation of prior node-wise and layer-wise sampling methods, and introduce two remedy: 1) use poison sampling distribution for debiasing, and 2) add back node-wise sampling probability to reduce per-node variance.\n\n2. Results show improvement against LADIES baseline.\n\nWEAKNESSES:\n1. The only two baselines this paper use is neighborhood sampling and LADIES. It's better to include more sampling baselines including FastGCN, GraphSAINT, etc.\n\n2. The paper cites \"Calibrate and Debias Layer-wise Sampling for Graph Convolutional Networks\", which provides several improvement to LADIES. It seems that the problem this paper tries to solve has many similar part of this paper. It's very important to discuss the difference and add the method (flat & debias) of this paper into evaluation.\n\n3. Regarding training speed & time cost, it's better to include a comparison on total time required for the model to converge. Currently the paper only includes iteration/s, but not how many iterations are needed. It's better to show the total time.\n\n4. It's highly recommended to add a psudo-code to show how to implement the proposed LABOR, to make readers understand the method easily.\n\nQUESTIONS:\nSee the concerns in the weakness part.\n\n\nAlso, it seems that the pdf format of the submitted paper does not support selecting & searching text, which makes it hard to find some part in the paper. Better to correct this part.\n\nEven after reading the paper, I still think the title of this paper is a bit misleading and unclear. I recommend the authors to reconsider this title and highlight your contributions in the title.\n\nLIMITATIONS:\nNo limitation provided in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel neighbor sampling (NS) method, called layer-neighbor sampling (LABOR), for training GNNs on large-scale graphs. Unlike the representative NS method, which samples neighbors for each node from a uniform distribution, LABOR samples neighbors for each node using Poisson sampling from a modified distribution. This modified distribution is optimized to match the variance of LABOR with that of NS for each single node. As a result, LABOR can reduce the number of sampled nodes by a large margin compared to NS. Experiments demonstrate the effectiveness of LABOR.\n\nSTRENGTHS:\nOriginality: The proposed method is novel in applying Poisson sampling and its effort to optimize a modified distribution. \n\nQuality: This paper falls within the borderline category, and introducing some in-depth analyses could improve its quality.\n\nClarity: While some unexplained notations and problematic statements exist, this paper is generally easy to follow.\n\nSignificance: Neighborhood explosion is an important problem in GNNs. The proposed method serves as a complement to existing neighbor sampling methods. However, it appears that the computation and memory complexity of the proposed method still exhibits exponential growth with respect to the depth of GNNs.\n\nWEAKNESSES:\n1. Including complexity analysis can help readers to understand how much the proposed method improves over baselines.\n\n2. Wall-clock time is a crucial concern in the efficient training of GNNs. Since the proposed method introduces additional time overhead during neighbor sampling, it may not be advantageous in wall-clock time, despite its superiority in batch size.\n\n3. Baselines for comparison are somewhat old and not state-of-the-art. Introducing some state-of-the-art baselines for comparison may be more convincing.\n\nQUESTIONS:\n1. Equation (8) appears problematic since the variance is independent of the number of sampled nodes, i.e., $\\tilde{d}_s$ or n. If I am wrong about this, please correct me. Considering Equation (8) is fundamental in their follow-up analysis, it is important to explain this.\n\n2. In the first row of Figure 2, the proposed method behaves strangely in the last figure (flickr). Is there any explanation for this phenomenon?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of improving the efficiency of GNN training. It first points out that the node-based sampling methods that sample independently ad recursively for each node suffer the neighbor explosion problem (NEP), while the layer-based sampling methods that sample for the whole layer collectively cannot guarantee the approximation of each vertex would be good enough. \n\nTo address these issues, this paper proposes the LABOR sampler which combines the advantages of node and layer-based sampling using Poisson Sampling. The main idea is to conduct neighbor sampling by letting individual vertices make correlated decisions while sampling their neighbors, and introducing a hyperparameter on each node to afftect the expected number of the sampled neighbors. The sampling algorithm would optimize over the sampling probabilities and the newly introduced hyperparameter, s.t. can sample the minimum number of vertices in an unbiased way. Experiments show that LABOR can reduce the sampled nodes and edges, and thus enjoys better efficiency.\n\nSTRENGTHS:\n1. The idea of letting the algorithm decide the minimum neighbors to be sampled makes sense and works well.\n2. According to the experimental results, the proposed method works well in reducing the sampled nodes and edges while maintaining the same-level performance as baselines.\n3. Code is provided in the supplementary.\n\nWEAKNESSES:\n1. It would be more clear if the authors can put the main algorithm in the main paper instead of in the supplementary.\n2. The authors only report the it/s (which can also indicates the run-time per iteration), it would be better if the authors can also report the total run-time or the number of training epochs needed to get converged.\n\nQUESTIONS:\nQ1: For Fig. 1, what is the ``cumulative # sampled nodes/edges``? Is it linear to the number of training iterations?\n\nQ2: How would different fanouts affect the performance?\n\nLIMITATIONS:\nPlease refer to \"Weakness\".", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}