Title: value: Layer-Neighbor Sampling --- Defusing Neighborhood Explosion in GNNs
Authors: value: Muhammed Fatih Balin
Umit Catalyurek
Venue: NeurIPS 2023
Keywords: value: Graph Neural Networks
Graph Sampling
GNN
Layer Sampling
Minibatch Training

Abstract: value: Graph Neural Networks (GNNs) have received significant attention recently, but training them at a large scale remains a challenge.
Mini-batch training coupled with sampling is used to alleviate this challenge.
However, existing approaches either suffer from the neighborhood explosion phenomenon or have suboptimal performance. 
To address these issues, we propose a new sampling algorithm called LAyer-neighBOR sampling (LABOR). 
It is designed to be a direct replacement for Neighbor Sampling (NS) with the same fanout hyperparameter while sampling up to 7 times fewer vertices, without sacrificing quality.
By design, the variance of the estimator of each vertex matches NS from the point of view of a single vertex.
Moreover, under the same vertex sampling budget constraints, LABOR converges 
faster than existing layer sampling approaches and can use up to 112 times larger batch sizes compared to NS.


Review discussion:
•	LABOR has a better estimator of sampling with lower variances than prior work LADIES, which might not have a vertex with its neighbor among all sampled vertices.
•	LABOR has better vertex/edge efficiency than NS.
As a result, LABOR can reach the same F1-score by using fewer sampled vertices under the same hyperparameter settings.
The experimental evaluation is not sufficiently detailed.
Some technical details are not clear.
Figure 1 shows that the vertices efficiency of LADIES/PLADIES might be higher than LABOR in Reddit dataset. Why?
While the paper addresses an important topic, it has several issues that need to be addressed before publication:
•	LABOR outperforms NS in a graph with a high average degree (same as the assumption in the NEP problem), such as the Reddit dataset (average degree 493) shown in Table 2.
However, when the average degree of the dataset is low (e.g., flickr: 10.09), the performance gap between LABOR and NS will be smaller.
•	LABOR is more often compared with NS in the paper, but LABOR is highly based on LADIES/layer sampling method.
For example, the vertices efficiency only compared to NS in section 4.2, but Figure 1 shows that the vertices efficiency of LADIES/PLADIES might be higher than LABOR in Reddit dataset.
•	Neighbor Sampling (NS) is an important comparison subject in this paper, but it is not detailed in both Introduction and Background. The background simply states, “Neighbor Sampling: The neighbor sampling approach was proposed by Hamilton et al. [2017]” with a citation to GraphSage.
Yet, section 3 describes that “Neighbor Sampling currently seems to be the most popular sampling approach.” The latter statement does not seem to refer to GraphSage.
•	The paper needs to report the improvement in the total training time explicitly, as the abstract only mentions “7 times fewer vertices” and “use up to 112 times larger batch sizes compared to NS”.
•	Table 2 does not give detailed hyperparameter settings.
For example, how were the hyperparameters obtained? what’s the range for hyperparameter sweeping? LADIES paper shows a 5 times total time difference by only adjusting hyperparameters – LADIES (64) and LADIES (512) on the Cora dataset.
Moreover, details need to be provided in line241-242 and Figure2’s caption – “The hyperparameters of LADIES and PLADIES were picked to roughly match LABOR-”
•	“same sampling budget” needs to be defined more clearly in Table 2.
No ethics review needed.
We would like to thank the reviewer for the valuable feedback.
For the question about why LADIES and PLADIES vertex efficiencies are higher than LABOR on reddit: Table 2 shows how many vertices and edges are
sampled in each layer. We are interested in the $|V^3|$ and $|E^2|$ columns as they give the scale of the x-axis for Figure 1. As one can see,
LADIES variants sample more than an order of magnitude larger number of edges. Reddit, having an average degree around 500, is a very dense graph.
So, LADIES variants sample a lot of edges, taking advantage of the denseness of reddit. LABOR, especially LABOR-0 not only tries to be vertex
efficient, but also edge efficient as the number of sampled edges affects the runtime.
So, LADIES variants outperform LABOR variants when it comes
to vertex efficiency on the Reddit dataset.
But even among LADIES variants, PLADIES that we propose is more vertex efficient.
So one can say that
our proposed methods still outperform existing baselines (PLADIES and LABOR vs LADIES and NS).
For the limitations:
* It is true that when the average degree is low (10.09), and the fanout is 10, then the difference between NS and LABOR is not much.
That is due to
the fact that at that point, more than half the nodes are using their full neighborhoods. (Sampling 10 random vertices out of $d_s \leq 10$ vertices
simply results in using the full neighborhood.) This is a limitation of all sampling algorithms and we will make this clear in a revised version
of our paper.
* For Section 4.2, it was not clear to us how to include LADIES in the comparison. This is due to the fact that LADIES hyperparameters directly affect
how many vertices should be sampled in a layer.
Thus, changing the batch size doesn't change how many vertices are sampled in the last layer.
However,
for NS and LABOR, the expected number of sampled vertices in each layer is a monotonic function of the batch size and the fanout hyperparameters.
So,
it simply did not make sense to include LADIES variants in Section 4.2.
Also, Section 4.1 already achieves the budget sampling comparison of LADIES
and LABOR that Section 4.2 achieves for NS and LABOR. This is because we chose the hyperparameters of LADIES variants to match LABOR-* in Section 4.1.
One can verify this by looking at Table 2, $|V^3|$,  $|V^2|$, $|V^1|$ and $|V^0|$ columns. Note that other works comparing against LADIES simply
chose a constant number of vertices to be sampled in each layer, and as a result, LADIES performed badly, but we chose to do as fair a comparison
as we can.
* We used Neighbor Sampling synonymously with the sampling approach proposed in the GraphSAGE paper.
All GNN frameworks present the sampling algorithm proposed in GraphSAGE as NeighborSampler. We are not trying to discredit the GraphSAGE paper at all,
simply use the more common name in the GNN frameworks. We will add more citations to GraphSAGE throughout the paper to make this more clear.
* Appendix A.8 has a study on wall-time convergence and hyperparameter optimization, including a GraphSAINT variant. There, LABOR beats both NS
and GraphSAINT when it comes to wall-time convergence.
* Table 2 gives all the hyperparameters you need to replicate our LADIES experiments. NS and LABOR simply used a fanout value of 10 while LADIES
hyperparameters were chosen to match LABOR-* . You can compare LABOR-* and LADIES along the $|V^3|$, $|V^2|$, $|V^1|$ columns to verify our claim.
Simply using 64 or 512 vertices per layer for LADIES simply doesn't work well.
That is why, LADIES accuracies in other papers are lacking. In our
paper, one can clearly see that when you choose hyperparameters appropriately and fairly, the test accuracies of different sampling methods are
equivalent. LADIES performs worse than other methods on the Flickr dataset, and this is due to the fact that original LADIES implementation is biased,
using sampling without replacement while the formulation in the paper is sampling with replacement. Since LADIES authors do not account for the bias
introduced in their implementation, it simply performs worse than the other unbiased sampling algorithms.
In the experiments, our goal was to keep everything equal except for what is being compared. In Section 4.1, we chose hyperparameters of different
samplers so that it is fair, and did not perform any hyperparameter tuning. From the reported test accuracies, you can see that the end result ended
up being the same. For Section 4.2, we kept all hyperparameters equal except for the batch size to fit a given vertex sampling budget.
Thank you for the clarifications, especially regarding the experiments.
1. The paper is well-written and clear.
2. Some analysis is provided. The experiments on some large graph datasets verify the advantages of the proposed method.
1. Only variance reduction analysis is provided. Theoretical convergence analysis or generalization analysis is lacking.
2. This paper mainly compares with LADIES, which is a work in 2019. I don't think it is still a state-of-the-art work about graph sampling that is appropriate for comparison. Some more recent works should be covered in experiments for comparison.
1. The proposed method allows a larger batch size. This can make convergence faster.
However, a larger batch size may not lead to a better generalization. This makes the statement of a faster convergence weaker.
There is no potential negative societal impact of this work.
No ethics review needed.
We would like to thank the reviewer for the valuable feedback.
For the first point, since LABOR is a combination of NS and LADIES methods, it wouldn't be hard to extend any proof for their
convergence to LABOR. We think that such a theoretical analysis wouldn't be the most interesting part of the paper as it is clear that LABOR should
have similar convergence guarantees compared to NS (verified empirically in our experimental section).
For the second point, we would like to refer the reviewer to read our main rebuttal. There, we explain why we chose our experimental
setting as is and chose to keep the discussion among the core sampling algorithms, on top of which many other sampling algorithms build.
When it comes to your questions, one can either keep the batch size the same and converge at the same speed as NS (Section 4.1) or use the fact
that we sample the same number of vertices with a larger batch size and converge faster than NS (Section 4.2). Section 4.2 just demonstrates
the different trade-offs one can make when it comes to choosing hyperparameters. The user can simply choose whatever batch size they want, and LABOR
will always (can be proven mathematically easily) sample fewer vertices and edges compared to NS. We do not recommend using large batch sizes for GNN training, only wanted to highlight the different sampler behaviors under various conditions.
Thank you for your feedback.
It is a pity that some theoretical analysis is not offered.
However, the empirical results look good. I don't think the baseline issue (my Weanknesses 2) is solved or clearly clarified. I prefer to keep my current score, but I decrease my confidence.
1. It systemetically studeis the limitation of prior node-wise and layer-wise sampling methods, and introduce two remedy: 1) use poison sampling distribution for debiasing, and 2) add back node-wise sampling probability to reduce per-node variance.
2. Results show improvement against LADIES baseline.
1. The only two baselines this paper use is neighborhood sampling and LADIES. It's better to include more sampling baselines including FastGCN, GraphSAINT, etc.
2. The paper cites "Calibrate and Debias Layer-wise Sampling for Graph Convolutional Networks", which provides several improvement to LADIES. It seems that the problem this paper tries to solve has many similar part of this paper. It's very important to discuss the difference and add the method (flat & debias) of this paper into evaluation.
3.
Regarding training speed & time cost, it's better to include a comparison on total time required for the model to converge. Currently the paper only includes iteration/s, but not how many iterations are needed. It's better to show the total time.
4. It's highly recommended to add a psudo-code to show how to implement the proposed LABOR, to make readers understand the method easily.
See the concerns in the weakness part.
Also, it seems that the pdf format of the submitted paper does not support selecting & searching text, which makes it hard to find some part in the paper. Better to correct this part.
Even after reading the paper, I still think the title of this paper is a bit misleading and unclear. I recommend the authors to reconsider this title and highlight your contributions in the title.
No limitation provided in the main paper.
No ethics review needed.
We would like to thank the reviewer for the valuable feedback. We will be responding to the weaknesses the reviewer pointed out below:
1. FastGCN is superseded by LADIES because FastGCN doesn't even consider the recursive structure of sampling and the previous layer's sampled
vertices. A preliminary comparison against GraphSAINT can be found in Appendix A.8.
As for the other sampling methods, they are orthogonal.
2. In the Calibrate and Debias paper, they propose a really expensive method to debias sampling with quadratic complexity. Our proposed Poisson
sampling approach and PLADIES method simply have linear complexity and solve the same problem. We further propose a novel way to combine layer and neighbor sampling methods and call it Layer-Neighbor Sampling (LABOR).
3. Appendix A.8 has a study focused on wall time to convergence and hyperparameter optimization. It shows that LABOR is faster than NS and GraphSAINT.
Also, Figure 1 is linear in the number of training iterations, and Figure 3 in Appendix A.4 has the x-axis as the number of training steps.
4. Appendix has Algorithm 1 which shows a high-level pseudocode for LABOR. We will move it to the main paper for the camera-ready version as we will have 1 more page allowance.
We will correct the PDF issue about selecting and searching.
We will consider alternative titles, and we would be open to recommendations.
1. I highly suggest authors discuss the difference between "Calibrate and Debias Layer-wise Sampling for Graph Convolutional Networks" more clearly, and add them into your paper (in the revised version). As NOT all readers are experts in this field and are familiar with these literatures.
2. Again, I think convergence time is very important (even more important than #steps) and shall be put in the main table, with all different configs. Different baselines have different time per step, so comparing #steps is not enough.
3. Current Alg 1 is still super unclear. E.g., how do you solve eq 14? what is each notation means? what is the output and how it's used for sampling? The current form is super hard to understand.
The reason I think "Defusing Neighborhood Explosion" is not a good title is that I think most existing layer-wise sampling does not have neighbor explosion issue?
As long as they control #nodes to be sampled per layer. Could the authors highlight your difference to these methods in the title? (Also "existing approaches either suffer from the neighborhood explosion phenomenon or have poor performance" seem to be a very strong criticism to other layer-dependent methods, but at least in table 2, I didn't see significant improvement to LADIES, and you didn't compare with "Calibrate and Debias Layer-wise Sampling for Graph Convolutional Networks")
1. We will revise our paper to include details of the difference between our proposed PLADIES algorithm and [1]. Let us talk about some of those
differences in the following 2 paragraphs:
Authors of [1] noticed the same
bias issue as us, and propose a way to "de-bias" sampling without replacement by computing actual sampling probabilities in quadratic time
(Mentioned in [1] Section 5.5).
However, PLADIES simply takes linear time because we use Poisson sampling, and with
Poisson sampling the end probability of being sampled is simply the probability of the Bernoulli trial of success, simply $\pi_t$.
When one uses
weighted sampling without replacement as in [1], the sampling probability is actually not the weights provided initially and [1] proposes a way to
compute the actual probabilities and a way to de-bias them in quadratic time. There is no convergence difference when it comes to PLADIES and the
de-bias procedure proposed in [1], only the computational complexity of sampling is different. They also propose a way to flatten the
probabilities and these flattened probabilities can be fed to PLADIES as well.
So the flattening approach [1] proposes is orthogonal to our work.
To give an example of why weighted sampling without replacement is biased, consider the probabilities [0.7, 0.2, 0.1]. Now, if we want to sample 2
items without replacement, then notice that the probability of the first item, 0.7, is more than 2 times the rest of the items (0.3).
Thus, the first
item needs to be sampled always.
But even if we always sample the first item, the second item is going to be either 0.2 or 0.1, the probability
of which one it is going to be is [2/3, 1/3].
Thus, in the end, the probability of sampling individual items is [1, 2/3, 1/3].
However, these
resulting probabilities are not linearly proportional to the probabilities we started with. [1] fixes this in quadratic time. We propose to use
Poisson Sampling instead. With Poisson sampling, we can have a constant $c$ to adjust how many items we want to sample. We want 2 items in
expectation, so $0.7c + 0.2c + 0.1c = 2$.
But the solution is $c=2$, and the
probability of sampling the first item becomes 1.4, which is not possible.
So, we need to solve for $c$ in the following equation:
$\min(0.7c, 1) + \min(0.2c, 1) + \min(0.1c, 1) = 2$. The solution is $c=10/3$, and this can be found in linear time. The resulting probabilities
become: $\min(0.7c, 1) = 1$, $\min(0.2c, 1) = 2/3$, $\min(0.1c, 1) = 1/3$. Because we use these adjusted probabilities during GNN aggregation, our
method is unbiased, and faster than [1] due to computational complexity difference.
2. Note that GNN training runtime depends on how many vertices or edges are processed.
Thus, Figure 1 portrays time to convergence in a
hardware-agnostic way by scaling the x-axis by the number of processed vertices or edges. All sampling algorithms compared have linear time
complexity in our paper. We will add a version of Figure 1 where the x-axis is the wall time.
3. We will improve the clarity of Algorithm 1 by adding more details about variables and how they are computed in the camera-ready version.
Section 3.2.2 and Equations 15, 16, and 17 describe one way of solving Equation 14 in linear time.
The output is the sampled set of edges $E'$ along with any computed edge weights $A'$
derived from the importance sampling probabilities to be used as edge weights during aggregation (Equation 4a or 4b) to make the estimation unbiased.
Table 4 in [1] compares two different configurations of LADIES, "LADIES" uses a fixed number of sampled vertices per layer (512, 512, 512)
and "LADIES (2)" when sampling 2 times more than the previous layer (512, 1024, 2048).
When we look at their results, it is clear to see that
LADIES (2) outperforms LADIES on all datasets. This means that LADIES doesn't work well with fixed numbers of vertices per layer and it doesn't solve
the neighborhood explosion issue by doing that because it performs poorly as we mentioned in our paper.
What we do in our experiments instead is to follow LABOR-* , when choosing the LADIES variants' number of sampled vertices per layer, as can be seen in
Table 2 of our paper.
As a result, it is clear to see that LADIES (PLADIES especially because it is unbiased) performs reliably well,
matching LABOR and NS variants in accuracy on all datasets.
Thus, our work
suggests how one should pick the hyperparameters of LADIES variants in a suitable manner, and that is by looking at LABOR-* rather than simply
doubling the number of sampled vertices per layer. We think this is important because the graph topology can be very different for various
applications. Before our work, how to choose the hyperparameters of LADIES in a manner that is robust to changing the underlying graph did not exist.
[1] Yifan Chen, Tianning Xu, Dilek Hakkani-Tur, Di Jin, Yun Yang, Ruoqing Zhu, "Calibrate and Debias Layer-wise Sampling for Graph Convolutional Networks", TMLR 2023
1. The mentioning of results of Table 4 in (1) is good to support your claim, and it shall be added into your paper with clear discussion (as it's related to your title and main argument).
Also I'm not pretty convinced a linear increase of sampling node shall be claimed as "explosion", as this is what we mainly call for exponential increasing.
2. LABOR definitely has many good properties and design, which I'd encourage authors use plain language to summarize and highlight in the intro.
3. (and again, I'd suggest authors not using such subjective words as "poorly" in a research paper, even looking at Table4 in (1), the avg. performance gap between LADIES(1) and (2) seems not that significant. You could simply report the relative performance improvement of your approach compared with prior baselines, as well as NS)
4. In general response, you mention for some other sampling methods like VR-GCN and shadow, NS could be replaced with LABOR. This is an interesting claim, as previously people thought those methods are coupled with NS, and not for Layer-Sampling. Could the  authors elaborate such possibility? (and again, would be helpful to be put in the paper) It's very important for the GNN community to accept that Layer-sampling is useful and flexible.
We appreciate the follow-up response!
Please do note that for "LADIES (2)", [512, 1024, 2048] is not a linear increase but an
exponential one, at layer $i$, they sample $512 * 2^{i}$ vertices. And it is not clear why one should stop with the exponent $2$, one can go ahead and
use $3$ or $4$, etc.
as long as there is an accuracy improvement.
Similarly, one can choose smaller/larger fanout values for LABOR and NS algorithms
if one wanted to change their iteration runtime, although the end accuracy will be slightly affected.
> 3. (and again, I'd suggest authors not using such subjective words as "poorly" in a research paper, even looking at Table4 in (1), the avg. performance gap between LADIES(1) and (2) seems not that significant. You could simply report the relative performance improvement of your approach compared with prior baselines, as well as NS)
To clarify what we had meant by the accuracy difference between "LADIES" and "LADIES (2)", Table 4 in the Calibrate and Debias paper has the following values:
|                 |   reddit      | ogbn-arxiv | ogbn-mag | ogbn-proteins| ogbn-products|
|--------------|----------------|----------------|----------------|------------------|-------------------|
|LADIES     | 73.86±0.17| 60.95±0.31| 24.79±0.48| 68.28±0.05| 52.97±1.11|
|LADIES (2)|88.34±0.11| 64.01±0.39| 28.59±0.39| 68.17±0.10| 65.24±0.40|
The accuracy difference between these two differs by up to 15\%.
So, we wanted to say that the difference is in fact significant. And it is not clear that LADIES (3) or LADIES (4) will not be even better.
We will revisit our introduction and highlight the good properties of our proposed sampler LABOR. We will also be careful with our wording per your
suggestions when talking about related work.
We would be more than happy to elaborate. Let us start with VR-GCN, and then talk about SHADOW and GNS.
VR-GCN uses NS to sample a fixed number of neighbors (usually 2), and for the
neighbors that weren't sampled, it uses their historical embeddings at each layer instead of recomputing them. If you look at our paper's
supplementary material, you will see that Appendix A.3 details how to modify LABOR so that it samples a fixed number of neighbors in linear time,
just like NS. If we replace NS in VR-GCN with LABOR-0 along with the method in A.3, then VR-GCN will run faster because there will be fewer vertices
sampled while still sampling 2 neighbors per vertex.
SHADOW [2] paper in its Appendix C describes how it extracts the subgraphs it uses during training. One option is to use an L-hop extract. It says
that one can use NS to sample the $L$-hop neighborhood of a given minibatch of vertices, which then is converted to a subgraph to be used in training.
Table 1 shows some experimental results with $L=2$, (shadow-SAGE 2-hop). If they had used LABOR-0 instead, their code would have run faster with no
noticeable convergence difference as we demonstrate in our paper.
For GNS, simply drop-in replacing LABOR-0 in place of NS will give us Global Layer-Neighbor (LABOR) Sampling. And we would expect similar
runtime improvements there as well. GNS keeps a cache of vertices that they try to sample a fixed number of vertices from, and they replenish the
cache periodically. If they can't find enough vertices in the cache, they sample from the rest of the graph. As you can see, there is nothing in this
algorithm that requires you to use NS, and LABOR-0 is designed as a drop-in replacement for NS.
Since LABOR-0 along with the method described in Appendix Section A.3 is almost indistinguishable from NS
(But it is still technically layer sampling and takes advantage of neighborhood overlaps and samples fewer vertices),
one can directly replace any occurrence of NS with
LABOR-0. It is hard to even write a test to distinguish the outputs of the two algorithms, statistical correlation testing is required.
However, extending other works and using LABOR-0 instead of NS is hard to fit into a single paper and we leave it as future work. We will include some
of this discussion in our camera-ready version.
[2] Zeng et al. Decoupling the Depth and Scope of Graph Neural Networks, arXiv:2201.07858
Thanks the authors' responses. Adding these discussion and descriptions for previous baselines are important to be updated.
I'd like to increase the score to "weak accept"
Originality: The proposed method is novel in applying Poisson sampling and its effort to optimize a modified distribution.
Quality: This paper falls within the borderline category, and introducing some in-depth analyses could improve its quality.
Clarity: While some unexplained notations and problematic statements exist, this paper is generally easy to follow.
Significance: Neighborhood explosion is an important problem in GNNs. The proposed method serves as a complement to existing neighbor sampling methods.
However, it appears that the computation and memory complexity of the proposed method still exhibits exponential growth with respect to the depth of GNNs.
1.
Including complexity analysis can help readers to understand how much the proposed method improves over baselines.
2. Wall-clock time is a crucial concern in the efficient training of GNNs. Since the proposed method introduces additional time overhead during neighbor sampling, it may not be advantageous in wall-clock time, despite its superiority in batch size.
3. Baselines for comparison are somewhat old and not state-of-the-art. Introducing some state-of-the-art baselines for comparison may be more convincing.
1. Equation (8) appears problematic since the variance is independent of the number of sampled nodes, i.e., $\tilde{d}_s$ or n. If I am wrong about this, please correct me. Considering Equation (8) is fundamental in their follow-up analysis, it is important to explain this.
2. In the first row of Figure 2, the proposed method behaves strangely in the last figure (flickr). Is there any explanation for this phenomenon?
yes
No ethics review needed.
We would like to thank the reviewer for the valuable feedback.
When it comes to computational complexity of sampling, let us note that we implemented all parts of LABOR using CUDA, fully parallelized. LABOR-0
sampling runtime is very similar to or even faster than NS.
When importance sampling is used, each iteration of optimizing sampling probabilities
takes linear time, basically an SpMV operation, which we accelerate using CUSparse. Our code is available in the supplementary material so you can
verify our claims by looking at the code. The computational complexity of forward-backward pass is simply linear in the number of vertices and edges
sampled, which are provided in Table 2.
Wall-clock time is simply proportional to the number of vertices and edges sampled if all algorithms are optimized as much as possible. Because
we did not have the time to optimize the LADIES variants as much as we did for our proposed sampled LABOR, we chose to scale the x-axis with the
cumulative number of sampled vertices or edges in Figure 1 (linear to the number of training steps and numbers reported in Table 2 $|V^3|$ and
$|E^2|$ columns) to make it as fair of a comparison as possible. A very optimized CUDA implementation for NS is already available in DGL and as you
can see LABOR is faster than NS even when it comes to iteration speed of training (Table 2 it/s column). Please see Appendix A.8 for a comparison focused on wall time to convergence.
As for the baselines, our appendix includes a preliminary comparison against the GraphSAINT method.
When it comes to most of the other
state-of-the-art samplers, our method is simply orthogonal to them.
Poisson sampling simply works by flipping unfair coins to determine whether an edge is sampled. If the probability to sample the edge $t \to s$ is
$\pi_t$, then the expected number of sampled vertices for $s$ will be $\sum_{t \to s} \pi_t$ and the expected number of sampled vertices for the
whole layer will be $n = \sum_{t \in N(S)} \pi_t$.
So the expected number of sampled vertices $n$ depends on the sum of values of $\pi_t$ in this case.
Figure 2, Flickr behavior is simply over fitting. In our experiments for Figure 2, we keep all hyperparameters the same except for the batch size.
LABOR can afford to use a larger batch size as reported in Table 3 under the same vertex sampling budget but simply quickly over fits in this
experiment. Our goal was to show that one can either sample fewer vertices and converge in a similar behavior to NS (Section 4.1) or increase the
batch size and converge faster than NS (Section 4.2). We do not recommend people to use very large batch sizes for GNN training, we simply wanted
to highlight the different behavior of sampling algorithms.
The authors have addressed most of my concerns.
Hence, I raise my score by one level.
But some concerns still remain unresolved. (1) The baselines used for comparison are somewhat outdated and not representative of the state-of-the-art methods.
While the authors mentioned that LABOR is orthogonal to most state-of-the-art samplers, all these samplers aim to improve the efficiency of training GNNs and can be directly compared against each other. If the authors can address this issue, I believe this paper will be more persuasive. (2) It is better to summarize the complexity of different methods in an independent table without considering specific datasets.
1. The idea of letting the algorithm decide the minimum neighbors to be sampled makes sense and works well.
2. According to the experimental results, the proposed method works well in reducing the sampled nodes and edges while maintaining the same-level performance as baselines.
3. Code is provided in the supplementary.
1. It would be more clear if the authors can put the main algorithm in the main paper instead of in the supplementary.
2. The authors only report the it/s (which can also indicates the run-time per iteration), it would be better if the authors can also report the total run-time or the number of training epochs needed to get converged.
Q1: For Fig. 1, what is the ``cumulative # sampled nodes/edges``? Is it linear to the number of training iterations?
Q2: How would different fanouts affect the performance?
Please refer to "Weakness".
No ethics review needed.
We would like to thank the reviewer for the valuable feedback. For the camera-ready version, we will have one more page available. We can move the
main algorithm to the main paper as you suggest.
Appendix A.4 has a figure where the x-axis is the number of training iterations. Appendix A.8 has a study about convergence with respect to wall time.
As for the Q1: It is indeed linear to the number of training iterations. The relationship is encoded in Table 2, $|V^3|$ and $|E^2|$
columns. We plotted Figure 1 this way as the computational complexity of a training step is $O(|V^3| + |E^2|)$ for a 3-layer GNN model as the other
layers simply have less work.
Also, some GPUs can be simply too fast, and the CPU bottlenecks in DGL can limit the empirical runtime improvements
despite LABOR sampling fewer vertices and edges. We captured the speed of an iteration in Table 1.
However, on larger-scale datasets where training
takes longer and larger batch sizes are used, and more expensive GNN models such as GAT are used, the benefits start becoming more apparent even
if the fastest GPUs are used for training. Runtime starts to follow the number of sampled vertices and edges in these cases. See Appendix A.6 and Table 5 in the supplementary material.
For Q2: Flickr dataset has an average degree of 10. If we increase the fanout, then all sampling methods in our experiments would start using the
full neighborhoods for almost all vertices. This would make all sampling methods give similar results.
However, for other datasets where we have room
to increase the fanout while still keeping things random, the difference between NS and LABOR and LABOR-i would grow even more.
When it comes to the
comparison with LADIES variants, we simply chose their hyperparameters so that it equals LABOR-* when it comes to the number of sampled vertices (Table 2) and would expect a comparatively similar convergence behavior across different samplers to the current fanout (10 for each layer)
experiments. We will add an ablation study
for different fanout experiments in the camera-ready version, and if time permits for the discussion period.
Thanks to the authors for your response. My questions are addressed. I would like to keep my original score 6.
We would like to thank all the reviewers for their valuable feedback.
Some reviewers question why we kept our main experimental evaluation to NS, LADIES and LABOR.
Firstly, Appendix A.8 has
some comparison against GraphSAINT in terms of time taken to converge to a given
accuracy target, and results shows that LABOR and NS outperform GraphSAINT.
As the graph becomes
larger and larger and the batch size stays fixed, we theorize that the following will happen: Subgraph sampling
methods similar in construction to GraphSAINT sample vertices/edges independently, thus as the graph size tends to
infinity, the number of edges in the sampled subgraph will tend to 0.
Hence, we think that GraphSAINT will not scale.
Cluster-GCN method is a biased method since it creates clusters and samples vertices
belonging to the same cluster together. In this paper, our goal was to investigate unbiased sampling methods and also
limit the problem space by assuming no knowledge of vertex embedding distribution so that we can focus the discussion
on the main limitations of LADIES and NS.
Also, we think that this kind of discussion can be left as future work as
our main focus was non-subgraph sampling methods.
Another subgraph sampling method is SHADOW (Decoupling the Depth and Scope of Graph Neural Networks), which uses
a given sampler such as NS or any LABOR variant as the EXTRACT method and generates an $L$-hop neighborhood.
Then,
they use the extracted subgraph and perform training with a model with more
than $L$ layers. Note that, even though it is a subgraph sampling method, it actually uses the $L$-hop neighborhood
of the given seed vertices, unlike GraphSAINT or Cluster-GCN.
Also, since it is a meta method, meaning that EXTRACT
can be any sampler, it can actually be enhanced by the use of LABOR-0 instead of NS algorithm they use in their
experiments, enjoying the same benefits we show in our paper.
One can see that the GNN sampling literature has chosen NS as their de facto sampling algorithm, our goal is to change
that to LABOR-0 (It is used as a direct drop-in) in this paper. Some example works, that we have cited in our paper,
almost explicitly using NS as their samplers:
* VR-GCN: Stochastic training of graph convolutional networks with variance reduction
* SHADOW: Decoupling the Depth and Scope of Graph Neural Networks
* GNS: Global Neighbor Sampling for Mixed CPU-GPU Training on Giant Graphs
* BS: Bandit samplers for training graph neural networks
* Thanos: A Biased Graph Neural Network Sampler with Near-Optimal Regret
All the papers above can be improved just by switching NS with LABOR-0, we are hoping that you are convinced, at this
point, that LABOR-0 outperforms NS in terms of efficiency while matching its quality.

Meta-review:
role: PC
data: comment: value: Reviewers acknowledged the contributions in terms of algorithm designs and experiments. The proposed algorithm has been shown to outperform the LADIES baseline and the proposed LABOR method can be incorporated into other layer-wise graph sampling algorithms to gain potential improvements. 

During the rebuttal phase, the authors have well addressed the concern raised by a high-confidence review (Reviewer qFRg) regarding the clarity and the comparison to LADIES. 

One potential issue could be the lack of rigorous theoretical analysis. This may not be as important as the empirical performance improvements (Reivewer QdGo seems partially agree with this), but I still suggest the authors at least conduct the analyses and add them to the revision.

Overall, this is a good paper that develops a new sampling method in training large graph neural networks, which accelerates the convergence and saves the memory cost. I would like to accept this paper. The authors need to carefully revise the paper according to the reviewers' suggestions.

Decision: Accept (poster)